Jul 28 01:25:49.033170 ip-10-0-138-218 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:25:49.470127 ip-10-0-138-218 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:49.470127 ip-10-0-138-218 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:25:49.470127 ip-10-0-138-218 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:49.470127 ip-10-0-138-218 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:25:49.470127 ip-10-0-138-218 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:49.470794 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.470209    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:25:49.478214 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478188    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:49.478214 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478206    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:49.478214 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478211    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:49.478214 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478217    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:49.478214 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478220    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478224    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478227    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478232    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478237    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478240    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478243    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478246    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478249    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478252    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478255    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478258    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478261    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478264    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478267    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478270    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478272    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478275    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478278    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:49.478409 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478281    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478284    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478287    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478289    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478312    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478317    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478321    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478325    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478328    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478331    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478334    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478337    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478340    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478343    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478347    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478351    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478354    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478359    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478362    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478365    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:49.478907 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478368    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478371    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478374    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478377    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478380    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478383    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478386    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478389    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478392    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478395    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478398    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478401    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478404    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478407    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478409    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478413    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478416    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478419    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478422    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478425    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:49.479456 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478427    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478430    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478435    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478438    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478440    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478443    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478446    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478450    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478453    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478456    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478460    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478463    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478466    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478469    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478472    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478475    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478478    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478481    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478485    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:49.480042 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478490    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478495    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478498    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478501    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478504    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478507    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478510    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478513    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478516    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478519    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478523    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478525    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478529    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478531    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478535    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478538    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.478541    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479296    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479303    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:49.480553 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479307    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479310    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479313    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479316    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479319    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479322    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479325    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479328    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479331    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479334    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479336    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479339    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479342    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479345    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479348    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479351    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479354    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479357    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479359    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479362    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:49.481063 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479365    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479368    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479370    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479374    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479377    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479379    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479382    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479385    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479387    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479390    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479393    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479395    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479399    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479402    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479404    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479407    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479410    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479413    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479416    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479418    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:49.481599 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479421    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479424    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479427    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479429    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479432    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479435    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479437    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479440    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479447    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479450    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479452    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479455    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479459    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479464    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479467    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479471    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479475    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479478    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479481    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479484    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:49.482146 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479488    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479492    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479496    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479500    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479504    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479507    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479510    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479513    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479518    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479521    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479525    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479528    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479531    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479534    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479537    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479540    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479543    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479546    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479549    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:49.482678 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479552    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479555    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479558    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479561    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479563    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479566    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479569    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479571    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479575    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479578    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479582    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479584    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479587    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479590    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479592    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479595    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479599    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.479602    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480029    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:49.483195 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480035    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480039    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480042    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480046    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480049    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480052    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480058    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480062    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480066    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480069    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480072    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480075    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480078    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480081    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480084    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480087    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480091    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480093    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480106    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480110    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:49.483804 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480113    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480116    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480119    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480121    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480124    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480127    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480130    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480132    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480135    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480138    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480141    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480144    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480147    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480150    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480152    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480155    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480158    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480161    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480163    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480166    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:49.484352 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480169    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480172    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480174    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480177    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480180    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480183    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480185    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480189    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480193    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480196    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480198    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480203    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480207    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480210    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480213    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480216    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480219    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480222    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480224    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480227    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:49.484929 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480230    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480232    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480235    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480238    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480240    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480243    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480246    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480249    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480251    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480254    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480257    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480260    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480263    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480265    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480268    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480271    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480273    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480276    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480279    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:49.485478 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480281    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480286    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480290    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480293    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480296    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480299    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480303    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480305    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480308    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480311    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480314    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480317    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480320    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480323    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480326    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480328    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480331    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480333    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.480336    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480402    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:25:49.485999 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480409    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480416    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480420    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480426    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480431    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480438    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480444    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480449    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480454    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480458    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480461    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480464    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480467    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480470    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480474    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480478    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480481    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480485    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480488    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480491    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480495    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480498    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480502    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:25:49.486528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480505    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480508    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480512    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480515    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480518    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480521    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480525    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480528    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480532    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480536    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480539    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480542    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480548    2577 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480551    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480556    2577 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480559    2577 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480562    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480566    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480569    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480572    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480575    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480580    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480583    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480586    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480590    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:25:49.487178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480593    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480597    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480600    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480603    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480606    2577 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480616    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480619    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480622    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480626    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480629    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480632    2577 flags.go:64] FLAG: --help="false"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480635    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480638    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480642    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480645    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480648    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480652    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480655    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480657    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480662    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480665    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480668    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480671    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480674    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:25:49.487872 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480677    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480680    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480683    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480686    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480691    2577 flags.go:64] FLAG: --lock-file=""
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480694    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480698    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480701    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480706    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480710    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480713    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480716    2577 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480719    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480722    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480725    2577 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480728    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480732    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480736    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480740    2577 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480743    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480746    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480749    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480752    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480756    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480759    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:25:49.488567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480762    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480769    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480774    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480777    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480780    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480783    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480788    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480791    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480795    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480798    2577 flags.go:64] FLAG: --port="10250"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480801    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480804    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ccf65cb87da9702d"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480811    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480814    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480817    2577 flags.go:64] FLAG: --register-node="true"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480821    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480827    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480831    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480834    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480837    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480841    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480844    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480847    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480850    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480853    2577 flags.go:64] FLAG: --runonce="false"
Jul 28 01:25:49.489245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480856    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480860    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480863    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480866    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480869    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480872    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480875    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480878    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480881    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480885    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480888    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480891    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480895    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480898    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480900    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480906    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480909    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480912    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480915    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480920    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480923    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480926    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480929    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480946    2577 flags.go:64] FLAG: --v="2"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480951    2577 flags.go:64] FLAG: --version="false"
Jul 28 01:25:49.489904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480956    2577 flags.go:64] FLAG: --vmodule=""
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480960    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.480964    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481113    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481117    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481120    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481123    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481126    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481129    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481132    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481135    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481138    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481141    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481144    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481147    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481149    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481153    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481156    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481159    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481162    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:49.490575 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481165    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481168    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481170    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481173    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481176    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481179    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481183    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481186    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481190    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481194    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481197    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481200    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481203    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481207    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481210    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481213    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481215    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481218    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481222    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:49.491164 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481225    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481229    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481233    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481236    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481238    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481241    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481244    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481247    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481249    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481253    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481256    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481259    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481262    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481265    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481268    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481270    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481273    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481276    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481278    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481283    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:49.491654 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481286    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481289    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481291    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481294    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481299    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481302    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481306    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481308    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481311    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481314    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481316    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481319    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481322    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481325    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481327    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481330    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481333    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481335    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481338    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:49.492206 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481340    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481345    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481348    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481350    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481353    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481356    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481359    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481361    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481364    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481367    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481370    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481375    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481379    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481382    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481385    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481388    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481391    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481394    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481397    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:49.492799 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481400    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481403    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481406    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.481409    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.482046    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.488997    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.489013    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.489027    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.489034    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489100    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489105    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489109    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489112    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489115    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489118    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:49.493332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489121    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489124    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489128    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489131    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489133    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489136    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489139    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489142    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489145    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489147    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489151    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489153    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489156    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489159    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489162    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489164    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489167    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489170    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489173    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489176    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:49.493743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489179    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489182    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489185    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489189    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489192    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489195    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489198    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489200    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489203    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489205    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489208    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489211    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489215    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489219    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489222    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489225    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489227    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489230    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489233    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489236    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:49.494367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489239    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489241    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489245    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489247    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489252    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489257    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489260    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489264    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489267    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489269    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489272    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489275    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489278    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489281    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489284    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489287    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489290    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489293    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489296    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:49.495114 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489299    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489301    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489304    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489307    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489310    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489313    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489316    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489319    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489321    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489324    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489327    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489330    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489332    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489335    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489338    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489341    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489344    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489347    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489349    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489352    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:49.495780 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489355    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489358    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489360    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489363    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489366    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489369    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489372    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489375    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489378    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489381    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489384    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489387    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489389    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489392    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.489398    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:49.496547 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489551    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489557    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489560    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489563    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489566    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489569    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489572    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489575    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489578    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489581    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489583    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489586    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489589    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489592    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489594    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489597    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489600    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489602    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489605    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489608    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:49.496998 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489611    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489613    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489617    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489619    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489622    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489625    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489628    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489631    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489634    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489637    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489640    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489643    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489645    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489648    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489651    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489654    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489657    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489661    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489665    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:49.497564 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489668    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489671    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489674    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489676    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489679    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489682    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489685    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489687    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489690    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489693    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489695    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489698    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489700    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489703    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489706    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489710    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489713    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489716    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489720    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:49.498241 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489723    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489725    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489728    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489731    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489734    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489737    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489740    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489743    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489745    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489748    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489751    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489753    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489756    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489759    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489762    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489764    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489767    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489770    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489773    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489776    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:49.498871 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489779    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489782    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489785    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489787    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489790    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489793    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489795    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489798    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489801    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489803    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489806    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489809    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489812    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489815    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489817    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489821    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489825    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489828    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489831    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489834    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:49.499722 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:49.489836    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.489841    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.489989    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.492492    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.494092    2577 server.go:1019] "Starting client certificate rotation"
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.494215    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.494276    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.498064    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:49.500403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.499886    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:49.513216 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.513197    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:25:49.515471 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.515454    2577 log.go:25] "Validated CRI v1 image API"
Jul 28 01:25:49.516308 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.516291    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:25:49.519198 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.519179    2577 fs.go:135] Filesystem UUIDs: map[1c065225-7b58-4cdf-9aed-d3c089613273:/dev/nvme0n1p4 6c52bb30-77f5-4a1c-8dc0-a52238cec707:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 01:25:49.519257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.519200    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:25:49.525641 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.525619    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:49.525751 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.525621    2577 manager.go:216] Machine: {Timestamp:2026-07-28 01:25:49.524135017 +0000 UTC m=+0.358882562 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091010 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29b52727c57f4e6e177e94ee19decb SystemUUID:ec29b527-27c5-7f4e-6e17-7e94ee19decb BootID:ef6a8921-be4f-4bfe-9bd6-60791c6de772 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:28:5d:ea:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:28:5d:ea:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:1d:d1:f0:6c:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:25:49.525751 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.525749    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:25:49.525895 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.525836    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:25:49.529641 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.529616    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:25:49.529829 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.529647    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:25:49.530678 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.530665    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:25:49.530750 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.530684    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:25:49.530750 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.530703    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:49.531910 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.531897    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:49.533269 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.533256    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:25:49.535280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535265    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:49.535419 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535407    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:25:49.535538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535458    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:25:49.535538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535479    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:25:49.535538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535502    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:25:49.535538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535516    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:25:49.535538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535531    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:25:49.535925 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535909    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:49.535925 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.535950    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:49.538333 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.538315    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:25:49.540269 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.540255    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:25:49.540315 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.540288    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:25:49.541889 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541874    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:25:49.541955 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541900    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:25:49.541955 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541912    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:25:49.541955 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541926    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541964    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541976    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.541991    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542002    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542022    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542034    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542046    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:25:49.542073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542055    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:25:49.542950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542917    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:25:49.542950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.542948    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:25:49.546400 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.546384    2577 server.go:1268] "Started kubelet"
Jul 28 01:25:49.546528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.546487    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:25:49.546637 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.546537    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:25:49.546637 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.546633    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:25:49.547188 ip-10-0-138-218 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:25:49.550812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.548189    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:25:49.550900 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.548178    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:25:49.551166 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.551138    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:25:49.551249 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.551217    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:25:49.553114 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.553095    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:25:49.553472 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.553449    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f87bg"
Jul 28 01:25:49.555254 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.554144    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-218.ec2.internal.18c64fb6f70a27bd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-218.ec2.internal,UID:ip-10-0-138-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-218.ec2.internal,},FirstTimestamp:2026-07-28 01:25:49.546260413 +0000 UTC m=+0.381007952,LastTimestamp:2026-07-28 01:25:49.546260413 +0000 UTC m=+0.381007952,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-218.ec2.internal,}"
Jul 28 01:25:49.556055 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.556038    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:25:49.558361 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.558334    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:25:49.558361 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.558348    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:49.558477 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.558454    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:49.559071 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559040    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:25:49.559168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559059    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:25:49.559168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559102    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:25:49.559359 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559189    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:25:49.559359 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559201    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:25:49.559909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559865    2577 factory.go:55] Registering systemd factory
Jul 28 01:25:49.559909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.559905    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:25:49.560122 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.560106    2577 factory.go:153] Registering CRI-O factory
Jul 28 01:25:49.560122 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.560123    2577 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:25:49.560226 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.560162    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:25:49.560226 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.560187    2577 factory.go:103] Registering Raw factory
Jul 28 01:25:49.560226 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.560205    2577 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:25:49.560856 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.560841    2577 manager.go:349] Starting recovery of all containers
Jul 28 01:25:49.562697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.562674    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f87bg"
Jul 28 01:25:49.565411 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.565388    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:25:49.565512 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.565471    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:25:49.569932 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.569915    2577 manager.go:354] Recovery completed
Jul 28 01:25:49.571788 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.571767    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 01:25:49.575005 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.574991    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.576969 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.576919    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.576969 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.576948    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.576969 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.576958    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.577493 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.577480    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:25:49.577493 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.577494    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:25:49.577579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.577511    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:49.577579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.577482    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.578330 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.578315    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.578387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.578342    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.578387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.578352    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.579346 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.579334    2577 policy_none.go:49] "None policy: Start"
Jul 28 01:25:49.579412 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.579350    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:25:49.579412 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.579362    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:25:49.579526 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.579455    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-218.ec2.internal.18c64fb6f8de455d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-218.ec2.internal,UID:ip-10-0-138-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-218.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-218.ec2.internal,},FirstTimestamp:2026-07-28 01:25:49.576938845 +0000 UTC m=+0.411686386,LastTimestamp:2026-07-28 01:25:49.576938845 +0000 UTC m=+0.411686386,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-218.ec2.internal,}"
Jul 28 01:25:49.580340 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.580329    2577 policy_none.go:47] "Start"
Jul 28 01:25:49.615646 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.615510    2577 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:25:49.615744 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.615670    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:25:49.615744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.615684    2577 server.go:87] "Starting device plugin registration server"
Jul 28 01:25:49.615951 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.615936    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:25:49.616049 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.616025    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:25:49.616178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.616155    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:25:49.616268 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.616261    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:25:49.616323 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.616271    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:25:49.616791 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.616772    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:25:49.616885 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.616820    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:49.689881 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.689853    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:25:49.691106 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.691088    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:25:49.691172 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.691131    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:25:49.691172 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.691154    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:25:49.691254 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.691185    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:25:49.693694 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.693674    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:49.717231 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.717208    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.718146 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.718130    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.718223 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.718159    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.718223 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.718170    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.718223 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.718192    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.728549 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.728533    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.728618 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.728553    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-218.ec2.internal\": node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:49.747612 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.747590    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:49.791920 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.791895    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"]
Jul 28 01:25:49.791999 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.791966    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.793048 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.793031    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.793130 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.793058    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.793130 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.793068    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.794316 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.794304    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.794442 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.794426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.794498 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.794457    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.795011 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.794995    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.795108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.795020    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.795108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.795049    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.795108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.795064    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.795108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.795029    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.795299 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.795127    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.796552 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.796537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.796613 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.796563    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:49.797250 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.797234    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:49.797325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.797260    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:49.797325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.797270    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:49.827664 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.827642    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-218.ec2.internal\" not found" node="ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.831310 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.831294    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-218.ec2.internal\" not found" node="ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.848228 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.848211    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:49.861274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.861251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c0e38832605850b3fb0e1ff07703dec-config\") pod \"kube-apiserver-proxy-ip-10-0-138-218.ec2.internal\" (UID: \"4c0e38832605850b3fb0e1ff07703dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.861360 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.861279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ad5e7d90023c47c9b3e6a26aadc8d7fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal\" (UID: \"ad5e7d90023c47c9b3e6a26aadc8d7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.861360 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.861296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e7d90023c47c9b3e6a26aadc8d7fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal\" (UID: \"ad5e7d90023c47c9b3e6a26aadc8d7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.949553 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:49.949465    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:49.961721 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.961685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ad5e7d90023c47c9b3e6a26aadc8d7fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal\" (UID: \"ad5e7d90023c47c9b3e6a26aadc8d7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.961863 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.961729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e7d90023c47c9b3e6a26aadc8d7fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal\" (UID: \"ad5e7d90023c47c9b3e6a26aadc8d7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.961863 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.961749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c0e38832605850b3fb0e1ff07703dec-config\") pod \"kube-apiserver-proxy-ip-10-0-138-218.ec2.internal\" (UID: \"4c0e38832605850b3fb0e1ff07703dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.961863 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.961772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ad5e7d90023c47c9b3e6a26aadc8d7fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal\" (UID: \"ad5e7d90023c47c9b3e6a26aadc8d7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.961863 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.961793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c0e38832605850b3fb0e1ff07703dec-config\") pod \"kube-apiserver-proxy-ip-10-0-138-218.ec2.internal\" (UID: \"4c0e38832605850b3fb0e1ff07703dec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:49.961863 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:49.961850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad5e7d90023c47c9b3e6a26aadc8d7fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal\" (UID: \"ad5e7d90023c47c9b3e6a26aadc8d7fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:50.049829 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:50.049776    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:50.134922 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.134886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:50.135069 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.134889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:50.149904 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:50.149876    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:50.250684 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:50.250596    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:50.357256 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:50.357219    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:50.459413 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:50.459370    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-218.ec2.internal\" not found"
Jul 28 01:25:50.475283 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.475260    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:50.493689 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.493669    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:25:50.493797 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.493780    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:50.493839 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.493825    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:50.543529 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.543476    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:50.558482 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.558457    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:50.558563 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.558481    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:50.568828 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.568804    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:50.569925 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.569898    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:20:50 +0000 UTC" deadline="2028-01-24 14:22:32.612761696 +0000 UTC"
Jul 28 01:25:50.569998 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.569926    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13092h56m42.042839755s"
Jul 28 01:25:50.570422 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.570408    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:50.572622 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.572609    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal"
Jul 28 01:25:50.584717 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.584695    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:50.595540 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.595519    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2j626"
Jul 28 01:25:50.603299 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.603277    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2j626"
Jul 28 01:25:50.813171 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:50.813155    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:51.061572 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.061469    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:51.537138 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.537052    2577 apiserver.go:52] "Watching apiserver"
Jul 28 01:25:51.546100 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.546068    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:25:51.546538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.546505    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-74w4r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal","openshift-multus/multus-chg9p","openshift-multus/network-metrics-daemon-qbb85","openshift-network-operator/iptables-alerter-l8jfw","openshift-ovn-kubernetes/ovnkube-node-ccfz9","kube-system/global-pull-secret-syncer-pqlq7","kube-system/konnectivity-agent-wthds","kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qlbrg","openshift-image-registry/node-ca-2thtx","openshift-multus/multus-additional-cni-plugins-sf8nb","openshift-network-diagnostics/network-check-target-fnv6c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"]
Jul 28 01:25:51.549182 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.549161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.549297 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.549244    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:25:51.551443 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.551412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:51.551549 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.551498    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:25:51.553835 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.553793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.556336 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.556216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:51.556336 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.556283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:25:51.558825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.558560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.558825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.558587    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.559000 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.558834    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:25:51.559201 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.559175    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:25:51.559360 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.559333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.559499 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.559477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wqv29\""
Jul 28 01:25:51.561038 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.561016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.564370 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.564035    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.564370 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.564221    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:25:51.564370 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.564266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k7tk6\""
Jul 28 01:25:51.564709 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.564652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.567017 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.566899    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.567136 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.567112    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.567838 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.567818    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:25:51.568257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568141    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:25:51.568257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568159    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:25:51.568257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.568257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:25:51.568257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568204    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.568549 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568474    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.568549 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.568488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hqvhw\""
Jul 28 01:25:51.569476 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.569602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-var-lib-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-etc-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-ovn\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-os-release\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khb88\" (UniqueName: \"kubernetes.io/projected/695e801e-7959-454a-9698-e492ff548233-kube-api-access-khb88\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-slash\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-node-log\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-cni-netd\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-cnibin\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-socket-dir-parent\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-kubelet\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/695e801e-7959-454a-9698-e492ff548233-cni-binary-copy\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-cni-bin\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg6zl\" (UniqueName: \"kubernetes.io/projected/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-kube-api-access-qg6zl\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:51.569966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37b9f52c-c910-42db-af8a-1ca46fc81799-iptables-alerter-script\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.569986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37b9f52c-c910-42db-af8a-1ca46fc81799-host-slash\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-log-socket\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-ovnkube-script-lib\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-cni-multus\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-etc-kubernetes\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-run-netns\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-hostroot\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-kubelet\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.570474 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-systemd-units\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-cni-bin\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/907860fa-083a-47ae-a4ad-dc7776e93632-dbus\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-system-cni-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mznjb\" (UniqueName: \"kubernetes.io/projected/37b9f52c-c910-42db-af8a-1ca46fc81799-kube-api-access-mznjb\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-systemd\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-ovnkube-config\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-env-overrides\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e75cf689-a822-4001-b7e2-472d72df64c2-ovn-node-metrics-cert\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-cni-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-netns\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxh5q\" (UniqueName: \"kubernetes.io/projected/e75cf689-a822-4001-b7e2-472d72df64c2-kube-api-access-jxh5q\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.570905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/907860fa-083a-47ae-a4ad-dc7776e93632-kubelet-config\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-k8s-cni-cncf-io\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571148 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-conf-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/695e801e-7959-454a-9698-e492ff548233-multus-daemon-config\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-multus-certs\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571248    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kpl6s\""
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571323    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571366    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-58d66\""
Jul 28 01:25:51.571930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.571709    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:25:51.574300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.573299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.578032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.576007    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:25:51.578032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.576056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pgqqx\""
Jul 28 01:25:51.578032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.576011    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:25:51.579488 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.579464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xr5b\""
Jul 28 01:25:51.579587 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.579495    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.579700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.579682    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.579838 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.579811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.582693 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.582676    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:25:51.582852 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.582812    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pbjmw\""
Jul 28 01:25:51.582938 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.582879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.583118 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.582710    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:25:51.583473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.583436    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:25:51.586003 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.585982    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-prmw9\""
Jul 28 01:25:51.586181 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.586155    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:25:51.586254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.586216    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:25:51.605195 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.605100    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:51 +0000 UTC" deadline="2028-01-23 01:37:38.935941334 +0000 UTC"
Jul 28 01:25:51.605195 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.605195    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13056h11m47.330749947s"
Jul 28 01:25:51.660684 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.660658    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:25:51.671318 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-k8s-cni-cncf-io\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-conf-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/695e801e-7959-454a-9698-e492ff548233-multus-daemon-config\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-node-log\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.671453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.671453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-var-lib-kubelet\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-os-release\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khb88\" (UniqueName: \"kubernetes.io/projected/695e801e-7959-454a-9698-e492ff548233-kube-api-access-khb88\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-slash\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cnibin\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-hosts-file\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgbm\" (UniqueName: \"kubernetes.io/projected/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-kube-api-access-mpgbm\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0e34270-2b92-4098-93e6-028f9fe3f1ee-serviceca\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-cnibin\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-socket-dir-parent\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-kubelet\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmc7\" (UniqueName: \"kubernetes.io/projected/f0e34270-2b92-4098-93e6-028f9fe3f1ee-kube-api-access-bmmc7\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-cni-bin\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.671734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg6zl\" (UniqueName: \"kubernetes.io/projected/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-kube-api-access-qg6zl\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-log-socket\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-ovnkube-script-lib\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-kubernetes\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-systemd\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-lib-modules\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-etc-kubernetes\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysctl-d\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-tuned\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-os-release\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.671997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-hostroot\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-kubelet\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-systemd-units\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-cni-bin\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/907860fa-083a-47ae-a4ad-dc7776e93632-dbus\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.672518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-system-cni-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mznjb\" (UniqueName: \"kubernetes.io/projected/37b9f52c-c910-42db-af8a-1ca46fc81799-kube-api-access-mznjb\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-systemd\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-device-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-sys-fs\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsgt\" (UniqueName: \"kubernetes.io/projected/854489b6-2601-4b26-9097-358478a0cc88-kube-api-access-5bsgt\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-cni-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-netns\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxh5q\" (UniqueName: \"kubernetes.io/projected/e75cf689-a822-4001-b7e2-472d72df64c2-kube-api-access-jxh5q\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysctl-conf\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-netns\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-kubelet\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/907860fa-083a-47ae-a4ad-dc7776e93632-dbus\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-etc-kubernetes\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-cni-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-conf-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.673463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-multus-socket-dir-parent\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-cni-bin\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-system-cni-dir\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-cni-bin\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-os-release\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-kubelet\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-systemd-units\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-systemd\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-log-socket\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-node-log\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-slash\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-k8s-cni-cncf-io\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-hostroot\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.672997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-run\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-cnibin\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-tmp\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.674357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqstm\" (UniqueName: \"kubernetes.io/projected/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-kube-api-access-fqstm\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-system-cni-dir\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/907860fa-083a-47ae-a4ad-dc7776e93632-kubelet-config\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/695e801e-7959-454a-9698-e492ff548233-multus-daemon-config\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/907860fa-083a-47ae-a4ad-dc7776e93632-kubelet-config\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-multus-certs\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-run-multus-certs\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-var-lib-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-etc-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-var-lib-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-ovnkube-script-lib\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-ovn\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-ovn\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-cni-netd\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-etc-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb4x\" (UniqueName: \"kubernetes.io/projected/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-kube-api-access-lsb4x\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.675275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-cni-netd\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-socket-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cni-binary-copy\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-tmp-dir\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0e34270-2b92-4098-93e6-028f9fe3f1ee-host\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.673926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/695e801e-7959-454a-9698-e492ff548233-cni-binary-copy\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37b9f52c-c910-42db-af8a-1ca46fc81799-iptables-alerter-script\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37b9f52c-c910-42db-af8a-1ca46fc81799-host-slash\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-registration-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-modprobe-d\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce65bb36-da1a-4be4-84cd-b01bc0fabece-konnectivity-ca\") pod \"konnectivity-agent-wthds\" (UID: \"ce65bb36-da1a-4be4-84cd-b01bc0fabece\") " pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-cni-multus\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-run-netns\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-host\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.676043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysconfig\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce65bb36-da1a-4be4-84cd-b01bc0fabece-agent-certs\") pod \"konnectivity-agent-wthds\" (UID: \"ce65bb36-da1a-4be4-84cd-b01bc0fabece\") " pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-ovnkube-config\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-env-overrides\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e75cf689-a822-4001-b7e2-472d72df64c2-ovn-node-metrics-cert\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/695e801e-7959-454a-9698-e492ff548233-cni-binary-copy\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37b9f52c-c910-42db-af8a-1ca46fc81799-host-slash\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-sys\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37b9f52c-c910-42db-af8a-1ca46fc81799-iptables-alerter-script\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/695e801e-7959-454a-9698-e492ff548233-host-var-lib-cni-multus\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.674659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-run-openvswitch\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.674703    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.674750    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.675012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-env-overrides\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.675064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e75cf689-a822-4001-b7e2-472d72df64c2-host-run-netns\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.675078    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:25:51.676892 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.675126    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:52.175103541 +0000 UTC m=+3.009851087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:51.677801 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.675154    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:25:52.175143464 +0000 UTC m=+3.009890995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:51.677801 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.675326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e75cf689-a822-4001-b7e2-472d72df64c2-ovnkube-config\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.679026 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.679002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e75cf689-a822-4001-b7e2-472d72df64c2-ovn-node-metrics-cert\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.682931 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.682908    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:51.682931 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.682937    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:51.682931 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.682952    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:51.683207 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:51.683016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:52.182999147 +0000 UTC m=+3.017746692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:51.684257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.684230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khb88\" (UniqueName: \"kubernetes.io/projected/695e801e-7959-454a-9698-e492ff548233-kube-api-access-khb88\") pod \"multus-chg9p\" (UID: \"695e801e-7959-454a-9698-e492ff548233\") " pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.685726 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.685703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg6zl\" (UniqueName: \"kubernetes.io/projected/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-kube-api-access-qg6zl\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:51.686328 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.686274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxh5q\" (UniqueName: \"kubernetes.io/projected/e75cf689-a822-4001-b7e2-472d72df64c2-kube-api-access-jxh5q\") pod \"ovnkube-node-ccfz9\" (UID: \"e75cf689-a822-4001-b7e2-472d72df64c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.686522 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.686496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mznjb\" (UniqueName: \"kubernetes.io/projected/37b9f52c-c910-42db-af8a-1ca46fc81799-kube-api-access-mznjb\") pod \"iptables-alerter-l8jfw\" (UID: \"37b9f52c-c910-42db-af8a-1ca46fc81799\") " pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.695949 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.695619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal" event={"ID":"ad5e7d90023c47c9b3e6a26aadc8d7fd","Type":"ContainerStarted","Data":"d60fde08aa7fdb7bf813eba429b4e0062eee83f4c70fdd69f0faddc1031f7a08"}
Jul 28 01:25:51.701347 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.701314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal" event={"ID":"4c0e38832605850b3fb0e1ff07703dec","Type":"ContainerStarted","Data":"f7ba9e089a6c7f7782383d73ea4551af5ea57588d5b76ead6f4f648be3731c40"}
Jul 28 01:25:51.775183 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-device-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.775183 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-sys-fs\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsgt\" (UniqueName: \"kubernetes.io/projected/854489b6-2601-4b26-9097-358478a0cc88-kube-api-access-5bsgt\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysctl-conf\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-device-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-run\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-sys-fs\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-tmp\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-run\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqstm\" (UniqueName: \"kubernetes.io/projected/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-kube-api-access-fqstm\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-system-cni-dir\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.775436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb4x\" (UniqueName: \"kubernetes.io/projected/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-kube-api-access-lsb4x\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-socket-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-system-cni-dir\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cni-binary-copy\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-tmp-dir\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0e34270-2b92-4098-93e6-028f9fe3f1ee-host\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-registration-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-modprobe-d\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce65bb36-da1a-4be4-84cd-b01bc0fabece-konnectivity-ca\") pod \"konnectivity-agent-wthds\" (UID: \"ce65bb36-da1a-4be4-84cd-b01bc0fabece\") " pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-host\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysconfig\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce65bb36-da1a-4be4-84cd-b01bc0fabece-agent-certs\") pod \"konnectivity-agent-wthds\" (UID: \"ce65bb36-da1a-4be4-84cd-b01bc0fabece\") " pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysctl-conf\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-sys\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-sys\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776067 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-var-lib-kubelet\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cnibin\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-registration-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-hosts-file\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-var-lib-kubelet\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgbm\" (UniqueName: \"kubernetes.io/projected/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-kube-api-access-mpgbm\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0e34270-2b92-4098-93e6-028f9fe3f1ee-serviceca\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-modprobe-d\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmc7\" (UniqueName: \"kubernetes.io/projected/f0e34270-2b92-4098-93e6-028f9fe3f1ee-kube-api-access-bmmc7\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-kubernetes\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-systemd\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-lib-modules\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysctl-d\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-tuned\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.776966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-os-release\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-tmp-dir\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-kubernetes\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-os-release\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-socket-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cnibin\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysctl-d\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-cni-binary-copy\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-lib-modules\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-systemd\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-hosts-file\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-host\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.775724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0e34270-2b92-4098-93e6-028f9fe3f1ee-host\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/854489b6-2601-4b26-9097-358478a0cc88-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.776861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-sysconfig\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.777281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce65bb36-da1a-4be4-84cd-b01bc0fabece-konnectivity-ca\") pod \"konnectivity-agent-wthds\" (UID: \"ce65bb36-da1a-4be4-84cd-b01bc0fabece\") " pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.777435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.777834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.777579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0e34270-2b92-4098-93e6-028f9fe3f1ee-serviceca\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.779040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.779018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce65bb36-da1a-4be4-84cd-b01bc0fabece-agent-certs\") pod \"konnectivity-agent-wthds\" (UID: \"ce65bb36-da1a-4be4-84cd-b01bc0fabece\") " pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.779255 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.779233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-etc-tuned\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.779565 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.779542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-tmp\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.786691 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.786639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgbm\" (UniqueName: \"kubernetes.io/projected/6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67-kube-api-access-mpgbm\") pod \"node-resolver-74w4r\" (UID: \"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67\") " pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.787568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.787080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmc7\" (UniqueName: \"kubernetes.io/projected/f0e34270-2b92-4098-93e6-028f9fe3f1ee-kube-api-access-bmmc7\") pod \"node-ca-2thtx\" (UID: \"f0e34270-2b92-4098-93e6-028f9fe3f1ee\") " pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.787568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.787465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqstm\" (UniqueName: \"kubernetes.io/projected/99fd4db0-3b3f-46ea-9b3e-5a17066478e0-kube-api-access-fqstm\") pod \"tuned-qlbrg\" (UID: \"99fd4db0-3b3f-46ea-9b3e-5a17066478e0\") " pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.787568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.787547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb4x\" (UniqueName: \"kubernetes.io/projected/2d0bd5ff-4e9c-4b42-a919-ce3ade437a55-kube-api-access-lsb4x\") pod \"multus-additional-cni-plugins-sf8nb\" (UID: \"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55\") " pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:51.788443 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.788420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsgt\" (UniqueName: \"kubernetes.io/projected/854489b6-2601-4b26-9097-358478a0cc88-kube-api-access-5bsgt\") pod \"aws-ebs-csi-driver-node-gtb98\" (UID: \"854489b6-2601-4b26-9097-358478a0cc88\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.868759 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.868721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-chg9p"
Jul 28 01:25:51.880040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.880020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l8jfw"
Jul 28 01:25:51.889853 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.889830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:25:51.896663 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.896642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98"
Jul 28 01:25:51.904497 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.904478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-74w4r"
Jul 28 01:25:51.909352 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.909334    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:51.914494 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.914478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:25:51.920168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.920151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qlbrg"
Jul 28 01:25:51.927980 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.927962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2thtx"
Jul 28 01:25:51.934794 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:51.934774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sf8nb"
Jul 28 01:25:52.124883 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.124802    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:52.180077 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.180035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:52.180266 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.180100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:52.180266 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.180167    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:52.180266 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.180227    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:52.180266 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.180240    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:25:53.180221904 +0000 UTC m=+4.014969429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:52.180451 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.180282    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:53.180264872 +0000 UTC m=+4.015012401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:52.280968 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.280929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:52.281149 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.281113    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:52.281149 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.281136    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:52.281149 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.281150    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:52.281294 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.281230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:53.281212243 +0000 UTC m=+4.115959794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:52.605640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.605530    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:51 +0000 UTC" deadline="2028-02-23 00:39:09.496283361 +0000 UTC"
Jul 28 01:25:52.605640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.605576    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13799h13m16.890710502s"
Jul 28 01:25:52.691416 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:52.691386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:52.691568 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:52.691522    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:25:52.827367 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.827332    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce65bb36_da1a_4be4_84cd_b01bc0fabece.slice/crio-14015229c5d0eeb3e7a1454bcfa1d847a8e0f74713ab5662f2a70d43a65df947 WatchSource:0}: Error finding container 14015229c5d0eeb3e7a1454bcfa1d847a8e0f74713ab5662f2a70d43a65df947: Status 404 returned error can't find the container with id 14015229c5d0eeb3e7a1454bcfa1d847a8e0f74713ab5662f2a70d43a65df947
Jul 28 01:25:52.828462 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.828385    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e34270_2b92_4098_93e6_028f9fe3f1ee.slice/crio-f6d6c6c1bc6f69be411a820950d48dca3010a47e62cab0cab39502d01f8215d8 WatchSource:0}: Error finding container f6d6c6c1bc6f69be411a820950d48dca3010a47e62cab0cab39502d01f8215d8: Status 404 returned error can't find the container with id f6d6c6c1bc6f69be411a820950d48dca3010a47e62cab0cab39502d01f8215d8
Jul 28 01:25:52.830067 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.829629    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75cf689_a822_4001_b7e2_472d72df64c2.slice/crio-ec8ce909be6da47feea4d5c0b2b7e5b70569837367e3394b2c6b6a88bc2567a8 WatchSource:0}: Error finding container ec8ce909be6da47feea4d5c0b2b7e5b70569837367e3394b2c6b6a88bc2567a8: Status 404 returned error can't find the container with id ec8ce909be6da47feea4d5c0b2b7e5b70569837367e3394b2c6b6a88bc2567a8
Jul 28 01:25:52.832167 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.832131    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0bd5ff_4e9c_4b42_a919_ce3ade437a55.slice/crio-1fa3b37addac2979b73c5a13f964f6b57e6eb4e227928aba02d348d78c5282ec WatchSource:0}: Error finding container 1fa3b37addac2979b73c5a13f964f6b57e6eb4e227928aba02d348d78c5282ec: Status 404 returned error can't find the container with id 1fa3b37addac2979b73c5a13f964f6b57e6eb4e227928aba02d348d78c5282ec
Jul 28 01:25:52.834681 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.834657    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e801e_7959_454a_9698_e492ff548233.slice/crio-e0529ad5815666b3afb78bb69df2bd730c555409a63e3a5f26fc6945b2cd7a21 WatchSource:0}: Error finding container e0529ad5815666b3afb78bb69df2bd730c555409a63e3a5f26fc6945b2cd7a21: Status 404 returned error can't find the container with id e0529ad5815666b3afb78bb69df2bd730c555409a63e3a5f26fc6945b2cd7a21
Jul 28 01:25:52.836917 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.836529    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fd4db0_3b3f_46ea_9b3e_5a17066478e0.slice/crio-adf0b056571bb8e1acfdd31e05595c6dad3cdf84cbafcf7dcc0269c4e1a02d60 WatchSource:0}: Error finding container adf0b056571bb8e1acfdd31e05595c6dad3cdf84cbafcf7dcc0269c4e1a02d60: Status 404 returned error can't find the container with id adf0b056571bb8e1acfdd31e05595c6dad3cdf84cbafcf7dcc0269c4e1a02d60
Jul 28 01:25:52.837726 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:25:52.837705    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b9f52c_c910_42db_af8a_1ca46fc81799.slice/crio-d1a5181708425641f1baf0085841ef9d68eb95756ecb0db5e86c71d3bdffb74d WatchSource:0}: Error finding container d1a5181708425641f1baf0085841ef9d68eb95756ecb0db5e86c71d3bdffb74d: Status 404 returned error can't find the container with id d1a5181708425641f1baf0085841ef9d68eb95756ecb0db5e86c71d3bdffb74d
Jul 28 01:25:53.189254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.189150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:53.189254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.189197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:53.189466 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.189304    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:53.189466 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.189342    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:53.189466 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.189368    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.189351057 +0000 UTC m=+6.024098582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:53.189466 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.189393    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.189377716 +0000 UTC m=+6.024125245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:53.291323 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.290710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:53.291323 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.290938    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:53.291323 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.290974    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:53.291323 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.290989    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:53.291323 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.291060    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.291036984 +0000 UTC m=+6.125784535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:53.695487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.694474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:53.695487 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.694607    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:25:53.695487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.694698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:53.695487 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:53.694790    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:25:53.719244 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.719132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-chg9p" event={"ID":"695e801e-7959-454a-9698-e492ff548233","Type":"ContainerStarted","Data":"e0529ad5815666b3afb78bb69df2bd730c555409a63e3a5f26fc6945b2cd7a21"}
Jul 28 01:25:53.729492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.729460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"ec8ce909be6da47feea4d5c0b2b7e5b70569837367e3394b2c6b6a88bc2567a8"}
Jul 28 01:25:53.736368 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.736334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wthds" event={"ID":"ce65bb36-da1a-4be4-84cd-b01bc0fabece","Type":"ContainerStarted","Data":"14015229c5d0eeb3e7a1454bcfa1d847a8e0f74713ab5662f2a70d43a65df947"}
Jul 28 01:25:53.739672 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.739640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal" event={"ID":"4c0e38832605850b3fb0e1ff07703dec","Type":"ContainerStarted","Data":"cdf1d19a3246071e687ed337370e2c5f238235806466472866d88e53e10010fb"}
Jul 28 01:25:53.742848 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.742468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-74w4r" event={"ID":"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67","Type":"ContainerStarted","Data":"af6d8b669bf8055dde7935a1519fbc055af0604f2ea5a8612698e5e43826ed49"}
Jul 28 01:25:53.762243 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.762213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l8jfw" event={"ID":"37b9f52c-c910-42db-af8a-1ca46fc81799","Type":"ContainerStarted","Data":"d1a5181708425641f1baf0085841ef9d68eb95756ecb0db5e86c71d3bdffb74d"}
Jul 28 01:25:53.764243 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.764219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlbrg" event={"ID":"99fd4db0-3b3f-46ea-9b3e-5a17066478e0","Type":"ContainerStarted","Data":"adf0b056571bb8e1acfdd31e05595c6dad3cdf84cbafcf7dcc0269c4e1a02d60"}
Jul 28 01:25:53.781767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.781735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerStarted","Data":"1fa3b37addac2979b73c5a13f964f6b57e6eb4e227928aba02d348d78c5282ec"}
Jul 28 01:25:53.787906 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.787820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2thtx" event={"ID":"f0e34270-2b92-4098-93e6-028f9fe3f1ee","Type":"ContainerStarted","Data":"f6d6c6c1bc6f69be411a820950d48dca3010a47e62cab0cab39502d01f8215d8"}
Jul 28 01:25:53.794280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.794250    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad5e7d90023c47c9b3e6a26aadc8d7fd" containerID="f97838ae8570a1528125e28959aa813226fe8fd69331e7d9ea2f4dafab8dda97" exitCode=0
Jul 28 01:25:53.794394 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.794336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal" event={"ID":"ad5e7d90023c47c9b3e6a26aadc8d7fd","Type":"ContainerDied","Data":"f97838ae8570a1528125e28959aa813226fe8fd69331e7d9ea2f4dafab8dda97"}
Jul 28 01:25:53.801801 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.801774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98" event={"ID":"854489b6-2601-4b26-9097-358478a0cc88","Type":"ContainerStarted","Data":"7c365d6a5eb6177d0d98c5e41f4a36c0776950c038ecfab93965396920cf9715"}
Jul 28 01:25:53.811942 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:53.811889    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal" podStartSLOduration=2.811872679 podStartE2EDuration="2.811872679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:53.757328498 +0000 UTC m=+4.592076047" watchObservedRunningTime="2026-07-28 01:25:53.811872679 +0000 UTC m=+4.646620227"
Jul 28 01:25:54.692251 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:54.692215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:54.692461 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:54.692357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:25:54.822444 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:54.822406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal" event={"ID":"ad5e7d90023c47c9b3e6a26aadc8d7fd","Type":"ContainerStarted","Data":"9c6b9a2b053ead5607c40289227091237ad2ef4ad06af6ed57b432abd38594c1"}
Jul 28 01:25:54.838996 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:54.838887    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-218.ec2.internal" podStartSLOduration=3.838867057 podStartE2EDuration="3.838867057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:54.838522916 +0000 UTC m=+5.673270467" watchObservedRunningTime="2026-07-28 01:25:54.838867057 +0000 UTC m=+5.673614608"
Jul 28 01:25:55.209674 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:55.209636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:55.209866 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:55.209698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:55.209953 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.209877    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:55.210018 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.209960    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:59.209939459 +0000 UTC m=+10.044687006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:55.210470 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.210361    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:55.210470 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.210413    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:25:59.210398685 +0000 UTC m=+10.045146216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:55.311239 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:55.310562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:55.311239 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.310747    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:55.311239 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.310773    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:55.311239 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.310788    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:55.311239 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.310861    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:59.310829869 +0000 UTC m=+10.145577399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:55.693976 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:55.693874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:55.694202 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.694027    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:25:55.694482 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:55.694461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:55.694598 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:55.694578    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:25:56.692296 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:56.692262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:56.692776 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:56.692421    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:25:57.691347 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:57.691312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:57.691529 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:57.691456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:25:57.691784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:57.691754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:57.691900 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:57.691875    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:25:58.691816 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:58.691779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:58.692282 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:58.691941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:25:59.242689 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:59.242655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:59.242891 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:59.242706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:25:59.242891 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.242774    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:59.242891 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.242798    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:59.242891 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.242833    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:26:07.242815138 +0000 UTC m=+18.077562676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:59.242891 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.242848    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:07.242841624 +0000 UTC m=+18.077589147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:59.343621 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:59.343585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:59.343798 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.343761    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:59.343798 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.343782    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:59.343798 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.343795    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:59.343997 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.343856    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:07.343838041 +0000 UTC m=+18.178585591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:59.692950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:59.692851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:25:59.693359 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.692980    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:25:59.693359 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:25:59.693035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:25:59.693359 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:25:59.693116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:00.691933 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:00.691904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:00.692096 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:00.692019    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:01.692793 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:01.692308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:01.692793 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:01.692432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:01.693324 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:01.692928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:01.693324 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:01.693078    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:02.691590 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:02.691552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:02.691800 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:02.691661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:03.692357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:03.692327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:03.692767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:03.692363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:03.692767 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:03.692442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:03.692767 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:03.692582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:04.692016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:04.691977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:04.692156 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:04.692091    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:05.692190 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:05.692159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:05.692650 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:05.692297    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:05.692650 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:05.692645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:05.692780 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:05.692750    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:06.692051 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:06.692018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:06.692255 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:06.692135    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:07.308630 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:07.308602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:07.308810 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:07.308643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:07.308810 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.308743    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:07.308810 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.308786    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:07.308810 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.308799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:26:23.308783587 +0000 UTC m=+34.143531114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:07.308978 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.308833    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:23.308818053 +0000 UTC m=+34.143565597 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:07.408989 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:07.408956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:07.409140 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.409075    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:07.409140 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.409088    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:07.409140 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.409098    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:07.409250 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.409144    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:23.409129981 +0000 UTC m=+34.243877504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:07.691961 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:07.691885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:07.691961 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:07.691931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:07.692173 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.692022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:07.692173 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:07.692158    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:08.692201 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:08.692165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:08.692670 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:08.692298    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:09.692965 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:09.692924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:09.693391 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:09.693035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:09.693391 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:09.693093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:09.693391 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:09.693221    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:10.692338 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:10.692301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:10.692504 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:10.692437    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:11.692418 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.692188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:11.693119 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:11.692536    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:11.693119 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.692229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:11.693119 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:11.692659    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:11.855344 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.855125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qlbrg" event={"ID":"99fd4db0-3b3f-46ea-9b3e-5a17066478e0","Type":"ContainerStarted","Data":"2459ed867981297c28286194e39711148b649eaf8e915d5f02548e50f298ddd9"}
Jul 28 01:26:11.856766 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.856736    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d0bd5ff-4e9c-4b42-a919-ce3ade437a55" containerID="a793e5fb872a8454d320294a5b3b3636cf26ce1f60f56926de15b708715289c5" exitCode=0
Jul 28 01:26:11.856879 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.856815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerDied","Data":"a793e5fb872a8454d320294a5b3b3636cf26ce1f60f56926de15b708715289c5"}
Jul 28 01:26:11.858640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.858617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2thtx" event={"ID":"f0e34270-2b92-4098-93e6-028f9fe3f1ee","Type":"ContainerStarted","Data":"d42a639de296be0c98f854e02d064da42867ba1a8138cf93e491d5785f5876c7"}
Jul 28 01:26:11.860543 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.860525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98" event={"ID":"854489b6-2601-4b26-9097-358478a0cc88","Type":"ContainerStarted","Data":"67e81fbf21821b9b76d3dae3ba5d82859218ca5e668ac51c5d2a5e6d99b18acc"}
Jul 28 01:26:11.862104 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.862079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-chg9p" event={"ID":"695e801e-7959-454a-9698-e492ff548233","Type":"ContainerStarted","Data":"0ba24ad9855d94cda14aff43771b55774d66f14bea47321852854261db9d1233"}
Jul 28 01:26:11.864284 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.864258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"d506891c2de071b2d39b82f5894a1792cba8edfcc9f2eb303c92b4a6c12c404c"}
Jul 28 01:26:11.864359 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.864327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"29a157a96b5cce129126fdbb271dc5b836e457d8a96cad1114ce7fd38821ff1e"}
Jul 28 01:26:11.866402 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.866383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wthds" event={"ID":"ce65bb36-da1a-4be4-84cd-b01bc0fabece","Type":"ContainerStarted","Data":"48aaf3bc478e2dd8aebefc7587b7e83b8b52a452c8451aadc74365c99b7f7e67"}
Jul 28 01:26:11.867838 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.867819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-74w4r" event={"ID":"6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67","Type":"ContainerStarted","Data":"654e01943fee7d3efa47cd52419feaa5efa5b840d26b8ab1df66974bd927d6cd"}
Jul 28 01:26:11.881988 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.881951    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qlbrg" podStartSLOduration=3.216956303 podStartE2EDuration="21.881939947s" totalImagesPullingTime="18.664983644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:11.881579899 +0000 UTC m=+22.716327446" watchObservedRunningTime="2026-07-28 01:26:11.881939947 +0000 UTC m=+22.716687515"
Jul 28 01:26:11.912422 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.912385    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wthds" podStartSLOduration=7.5025531579999996 podStartE2EDuration="21.912372523s" totalImagesPullingTime="14.409819365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:11.897627623 +0000 UTC m=+22.732375169" watchObservedRunningTime="2026-07-28 01:26:11.912372523 +0000 UTC m=+22.747120069"
Jul 28 01:26:11.912585 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.912562    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2thtx" podStartSLOduration=3.242666283 podStartE2EDuration="21.912555524s" totalImagesPullingTime="18.669889241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:11.912146421 +0000 UTC m=+22.746893977" watchObservedRunningTime="2026-07-28 01:26:11.912555524 +0000 UTC m=+22.747303069"
Jul 28 01:26:11.929784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.929751    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-chg9p" podStartSLOduration=3.212470481 podStartE2EDuration="21.929740329s" totalImagesPullingTime="18.717269848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:11.929384339 +0000 UTC m=+22.764131884" watchObservedRunningTime="2026-07-28 01:26:11.929740329 +0000 UTC m=+22.764487875"
Jul 28 01:26:11.944803 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:11.944772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-74w4r" podStartSLOduration=3.290516186 podStartE2EDuration="21.944761323s" totalImagesPullingTime="18.654245137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:11.944631698 +0000 UTC m=+22.779379243" watchObservedRunningTime="2026-07-28 01:26:11.944761323 +0000 UTC m=+22.779508868"
Jul 28 01:26:12.691759 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:12.691725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:12.691980 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:12.691837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:12.873651 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:12.873617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"93224c72bf0b035e8cbef3a60c886a7e18cd0ac04709afddca0780a2f7f7b833"}
Jul 28 01:26:12.874345 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:12.873701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"aa8c6c6ab1781dd6b0c38c2a21a85b1ec8c40df037784cf96a34294533c4e183"}
Jul 28 01:26:12.874345 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:12.873744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"dc52459cf571848ed598e1a488f1d5ddc713463348e9df16a12168c847be153a"}
Jul 28 01:26:12.874345 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:12.873784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"199714069da7f646b18a98193b0e6fb37cd58bdbc91e33c371521a3c48ae5bee"}
Jul 28 01:26:13.181023 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.180997    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:26:13.633270 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.633134    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:26:13.181018631Z","UUID":"2947306c-3bfb-43f7-b077-c7a369462238","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:26:13.634814 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.634792    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:26:13.634960 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.634835    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:26:13.692093 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.692067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:13.692236 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.692065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:13.692236 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:13.692222    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:13.692350 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:13.692270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:13.878309 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.878271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98" event={"ID":"854489b6-2601-4b26-9097-358478a0cc88","Type":"ContainerStarted","Data":"8adbc2ddd86e48b5b7c77f06705fa9603587fabe949bada717c9994bc3eeb9e8"}
Jul 28 01:26:13.880258 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.880216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l8jfw" event={"ID":"37b9f52c-c910-42db-af8a-1ca46fc81799","Type":"ContainerStarted","Data":"becad0405d88735c073137973716e7dd0ee15ffe2308e1aa5eb9e8c92f338304"}
Jul 28 01:26:13.899571 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:13.899491    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l8jfw" podStartSLOduration=5.241858677 podStartE2EDuration="23.899478002s" totalImagesPullingTime="18.657619325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:13.898936086 +0000 UTC m=+24.733683633" watchObservedRunningTime="2026-07-28 01:26:13.899478002 +0000 UTC m=+24.734225549"
Jul 28 01:26:14.467178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:14.467102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:26:14.691507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:14.691477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:14.691750 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:14.691608    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:14.887224 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:14.887182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"73b456c1127f7a60472e47a5ad7f6950f30eb9a27f81fa6057024f995783d57a"}
Jul 28 01:26:14.889551 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:14.889523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98" event={"ID":"854489b6-2601-4b26-9097-358478a0cc88","Type":"ContainerStarted","Data":"b31d22cc1c8850221ca5c16ae45fd34f1c73b1b812522887e511f5d0105df591"}
Jul 28 01:26:14.910663 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:14.910620    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtb98" podStartSLOduration=4.056387966 podStartE2EDuration="24.910607286s" totalImagesPullingTime="20.85421932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:14.910291075 +0000 UTC m=+25.745038621" watchObservedRunningTime="2026-07-28 01:26:14.910607286 +0000 UTC m=+25.745354865"
Jul 28 01:26:15.692271 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:15.692080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:15.692457 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:15.692101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:15.692457 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:15.692401    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:15.692457 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:15.692427    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:16.023150 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.023123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:26:16.023647 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.023633    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:26:16.691692 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.691624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:16.691815 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:16.691719    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:16.896959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.896922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" event={"ID":"e75cf689-a822-4001-b7e2-472d72df64c2","Type":"ContainerStarted","Data":"806fe676b2e68e0cef41367ef55074f331c34ea7080cc23e2dcd72a7f853bee4"}
Jul 28 01:26:16.897134 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.897116    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:26:16.897387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.897190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:26:16.897387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.897238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:26:16.898998 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.898900    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d0bd5ff-4e9c-4b42-a919-ce3ade437a55" containerID="e6c34d6fdb907fcbdd67cd4d3116e8974254b9221b2c61900defbbdeaca40a29" exitCode=0
Jul 28 01:26:16.899106 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.898985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerDied","Data":"e6c34d6fdb907fcbdd67cd4d3116e8974254b9221b2c61900defbbdeaca40a29"}
Jul 28 01:26:16.899754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.899679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wthds"
Jul 28 01:26:16.913028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.913008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:26:16.913158 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.913147    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:26:16.925537 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:16.925500    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9" podStartSLOduration=8.196358447 podStartE2EDuration="26.925485878s" totalImagesPullingTime="18.729127431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:16.925238235 +0000 UTC m=+27.759985780" watchObservedRunningTime="2026-07-28 01:26:16.925485878 +0000 UTC m=+27.760233424"
Jul 28 01:26:17.692527 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:17.692454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:17.693316 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:17.692572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:17.693316 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:17.692583    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:17.693316 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:17.692687    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:17.693316 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:17.692753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:17.693316 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:17.692820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:18.484073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.484047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pqlq7"]
Jul 28 01:26:18.484265 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.484141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:18.484265 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:18.484233    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:18.487483 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.487455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qbb85"]
Jul 28 01:26:18.487587 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.487564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:18.487693 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:18.487673    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:18.488090 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.488066    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fnv6c"]
Jul 28 01:26:18.488158 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.488146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:18.488243 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:18.488220    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:18.905706 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.905625    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d0bd5ff-4e9c-4b42-a919-ce3ade437a55" containerID="44e1111d33ea86c129cfc28d0834a876f953eb5dba82be02709c7befc78b96d6" exitCode=0
Jul 28 01:26:18.906149 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:18.905715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerDied","Data":"44e1111d33ea86c129cfc28d0834a876f953eb5dba82be02709c7befc78b96d6"}
Jul 28 01:26:19.692592 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:19.692562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:19.692781 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:19.692643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:19.692781 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:19.692661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:19.692781 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:19.692764    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:20.692106 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:20.692039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:20.692422 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:20.692139    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:20.911184 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:20.911154    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d0bd5ff-4e9c-4b42-a919-ce3ade437a55" containerID="12e67f0f93bfafebeb609394c5b8747ea20adbfe7f8e1e867a04c75ab2a17735" exitCode=0
Jul 28 01:26:20.911184 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:20.911188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerDied","Data":"12e67f0f93bfafebeb609394c5b8747ea20adbfe7f8e1e867a04c75ab2a17735"}
Jul 28 01:26:21.691681 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:21.691642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:21.691880 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:21.691787    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:21.691880 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:21.691818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:21.691976 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:21.691938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:22.691596 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:22.691571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:22.692109 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:22.691674    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:23.328833 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:23.328795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:23.329028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:23.328841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:23.329028 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.328965    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:23.329028 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.329013    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:23.329157 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.329032    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:26:55.32901526 +0000 UTC m=+66.163762804 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:23.329157 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.329064    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret podName:907860fa-083a-47ae-a4ad-dc7776e93632 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:55.329048737 +0000 UTC m=+66.163796281 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret") pod "global-pull-secret-syncer-pqlq7" (UID: "907860fa-083a-47ae-a4ad-dc7776e93632") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:23.429905 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:23.429808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:23.430048 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.429987    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:23.430048 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.430011    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:23.430048 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.430023    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zfp2f for pod openshift-network-diagnostics/network-check-target-fnv6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:23.430173 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.430080    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f podName:ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:55.430060949 +0000 UTC m=+66.264808476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zfp2f" (UniqueName: "kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f") pod "network-check-target-fnv6c" (UID: "ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:23.691794 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:23.691722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:23.691794 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:23.691756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:23.692360 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.691845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pqlq7" podUID="907860fa-083a-47ae-a4ad-dc7776e93632"
Jul 28 01:26:23.692360 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:23.691945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnv6c" podUID="ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462"
Jul 28 01:26:24.691424 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:24.691388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:24.691610 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:24.691525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbb85" podUID="fb8c2454-c3c5-4499-a2d8-1a5cfe353eba"
Jul 28 01:26:25.474988 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.474962    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-218.ec2.internal" event="NodeReady"
Jul 28 01:26:25.475424 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.475078    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:26:25.528954 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.528899    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j7v6r"]
Jul 28 01:26:25.530534 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.530520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.533050 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.533029    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:26:25.533140 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.533053    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:26:25.533292 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.533276    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h497g\""
Jul 28 01:26:25.533369 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.533304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:26:25.545168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.545145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7v6r"]
Jul 28 01:26:25.635104 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.635073    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jpzzq"]
Jul 28 01:26:25.636748 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.636734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.639776 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.639754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:26:25.640063 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.640047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:26:25.640260 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.640246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mh7n2\""
Jul 28 01:26:25.647662 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.647642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764abbf1-6c23-4e88-adca-d80b035509ce-cert\") pod \"ingress-canary-j7v6r\" (UID: \"764abbf1-6c23-4e88-adca-d80b035509ce\") " pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.647752 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.647691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spm7\" (UniqueName: \"kubernetes.io/projected/764abbf1-6c23-4e88-adca-d80b035509ce-kube-api-access-6spm7\") pod \"ingress-canary-j7v6r\" (UID: \"764abbf1-6c23-4e88-adca-d80b035509ce\") " pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.650061 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.650039    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jpzzq"]
Jul 28 01:26:25.691685 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.691655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:25.691685 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.691674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:25.694362 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.694341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4mq45\""
Jul 28 01:26:25.694487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.694377    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:25.694487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.694407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:25.694487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.694380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:25.748180 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.748160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764abbf1-6c23-4e88-adca-d80b035509ce-cert\") pod \"ingress-canary-j7v6r\" (UID: \"764abbf1-6c23-4e88-adca-d80b035509ce\") " pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.748303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.748187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6df3021-0059-41bc-be13-3067ea24b062-tmp-dir\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.748303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.748213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df3021-0059-41bc-be13-3067ea24b062-config-volume\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.748303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.748248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.748303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.748291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spm7\" (UniqueName: \"kubernetes.io/projected/764abbf1-6c23-4e88-adca-d80b035509ce-kube-api-access-6spm7\") pod \"ingress-canary-j7v6r\" (UID: \"764abbf1-6c23-4e88-adca-d80b035509ce\") " pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.748505 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.748332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcd8\" (UniqueName: \"kubernetes.io/projected/b6df3021-0059-41bc-be13-3067ea24b062-kube-api-access-tgcd8\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.752534 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.752515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/764abbf1-6c23-4e88-adca-d80b035509ce-cert\") pod \"ingress-canary-j7v6r\" (UID: \"764abbf1-6c23-4e88-adca-d80b035509ce\") " pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.756607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.756590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spm7\" (UniqueName: \"kubernetes.io/projected/764abbf1-6c23-4e88-adca-d80b035509ce-kube-api-access-6spm7\") pod \"ingress-canary-j7v6r\" (UID: \"764abbf1-6c23-4e88-adca-d80b035509ce\") " pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.839619 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.839559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7v6r"
Jul 28 01:26:25.848825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.848799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.848955 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.848874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcd8\" (UniqueName: \"kubernetes.io/projected/b6df3021-0059-41bc-be13-3067ea24b062-kube-api-access-tgcd8\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.848955 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.848921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6df3021-0059-41bc-be13-3067ea24b062-tmp-dir\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.849061 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:25.848957    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:25.849061 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.848965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df3021-0059-41bc-be13-3067ea24b062-config-volume\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.849061 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:25.849024    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls podName:b6df3021-0059-41bc-be13-3067ea24b062 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:26.348998865 +0000 UTC m=+37.183746389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls") pod "dns-default-jpzzq" (UID: "b6df3021-0059-41bc-be13-3067ea24b062") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:25.849604 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.849583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6df3021-0059-41bc-be13-3067ea24b062-tmp-dir\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.849653 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.849619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6df3021-0059-41bc-be13-3067ea24b062-config-volume\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.858156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.858130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcd8\" (UniqueName: \"kubernetes.io/projected/b6df3021-0059-41bc-be13-3067ea24b062-kube-api-access-tgcd8\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:25.927886 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.927709    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d0bd5ff-4e9c-4b42-a919-ce3ade437a55" containerID="ec3b2d99c87de38da0d5cc0a616b90d6307b84d5b53e223cfd8870eb486b363a" exitCode=0
Jul 28 01:26:25.928002 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.927789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerDied","Data":"ec3b2d99c87de38da0d5cc0a616b90d6307b84d5b53e223cfd8870eb486b363a"}
Jul 28 01:26:25.974349 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:25.974319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7v6r"]
Jul 28 01:26:25.977703 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:25.977672    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764abbf1_6c23_4e88_adca_d80b035509ce.slice/crio-0d92a1d026e78657e676678448be93f19fc593e405a98031b3f44763581bea43 WatchSource:0}: Error finding container 0d92a1d026e78657e676678448be93f19fc593e405a98031b3f44763581bea43: Status 404 returned error can't find the container with id 0d92a1d026e78657e676678448be93f19fc593e405a98031b3f44763581bea43
Jul 28 01:26:26.353756 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.353710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:26.353963 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:26.353889    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:26.354007 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:26.353961    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls podName:b6df3021-0059-41bc-be13-3067ea24b062 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:27.353940516 +0000 UTC m=+38.188688059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls") pod "dns-default-jpzzq" (UID: "b6df3021-0059-41bc-be13-3067ea24b062") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:26.692018 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.691920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:26.694930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.694909    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:26.695016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.694913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5299b\""
Jul 28 01:26:26.934027 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.933985    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d0bd5ff-4e9c-4b42-a919-ce3ade437a55" containerID="ad9b86d8efd2edda9e9324e6ec934d4d367fcb772aec8aceb3b9f0ad7f87707e" exitCode=0
Jul 28 01:26:26.934227 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.934067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerDied","Data":"ad9b86d8efd2edda9e9324e6ec934d4d367fcb772aec8aceb3b9f0ad7f87707e"}
Jul 28 01:26:26.935496 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:26.935468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7v6r" event={"ID":"764abbf1-6c23-4e88-adca-d80b035509ce","Type":"ContainerStarted","Data":"0d92a1d026e78657e676678448be93f19fc593e405a98031b3f44763581bea43"}
Jul 28 01:26:27.361517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:27.361482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:27.361691 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:27.361642    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:27.361740 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:27.361724    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls podName:b6df3021-0059-41bc-be13-3067ea24b062 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:29.361703906 +0000 UTC m=+40.196451436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls") pod "dns-default-jpzzq" (UID: "b6df3021-0059-41bc-be13-3067ea24b062") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:27.941807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:27.941768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" event={"ID":"2d0bd5ff-4e9c-4b42-a919-ce3ade437a55","Type":"ContainerStarted","Data":"d0bef9f3a82fb72f02866b926d2d717d7f0647e18e567f1928957cec5e070a76"}
Jul 28 01:26:27.967804 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:27.967403    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sf8nb" podStartSLOduration=8.269352171 podStartE2EDuration="37.967381783s" totalImagesPullingTime="29.698029612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:27.965290512 +0000 UTC m=+38.800038087" watchObservedRunningTime="2026-07-28 01:26:27.967381783 +0000 UTC m=+38.802129330"
Jul 28 01:26:28.946349 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:28.946166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7v6r" event={"ID":"764abbf1-6c23-4e88-adca-d80b035509ce","Type":"ContainerStarted","Data":"cc89940bc3aa342e1fdc69e111e35f739bb80c118b4ca7d862f6bd61c5ab5a55"}
Jul 28 01:26:28.963260 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:28.963210    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j7v6r" podStartSLOduration=1.657920933 podStartE2EDuration="3.963189663s" totalImagesPullingTime="2.30526873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:28.96208028 +0000 UTC m=+39.796827826" watchObservedRunningTime="2026-07-28 01:26:28.963189663 +0000 UTC m=+39.797937209"
Jul 28 01:26:29.377882 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:29.377809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:29.378067 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:29.377966    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:29.378067 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:29.378029    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls podName:b6df3021-0059-41bc-be13-3067ea24b062 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:33.378014494 +0000 UTC m=+44.212762018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls") pod "dns-default-jpzzq" (UID: "b6df3021-0059-41bc-be13-3067ea24b062") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:33.404114 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:33.404078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:33.404490 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:33.404200    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:33.404490 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:33.404260    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls podName:b6df3021-0059-41bc-be13-3067ea24b062 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:41.404245213 +0000 UTC m=+52.238992737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls") pod "dns-default-jpzzq" (UID: "b6df3021-0059-41bc-be13-3067ea24b062") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:37.809054 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.809021    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"]
Jul 28 01:26:37.849649 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.849625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"]
Jul 28 01:26:37.849788 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.849656    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"]
Jul 28 01:26:37.849788 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.849731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:37.851915 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.851895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 01:26:37.852443 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.852426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.852508 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.852438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 01:26:37.852508 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.852445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4tsfg\""
Jul 28 01:26:37.852968 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.852954    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.863199 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.863180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"]
Jul 28 01:26:37.863297 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.863272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:37.866969 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.866952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.867060 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.866988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.867122 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.867108    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-szd96\""
Jul 28 01:26:37.867810 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.867794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 01:26:37.870269 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.870250    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 01:26:37.928707 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.928662    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"]
Jul 28 01:26:37.949212 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.949188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"
Jul 28 01:26:37.949212 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.949195    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-6k54l"]
Jul 28 01:26:37.951540 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.951523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jj5l6\""
Jul 28 01:26:37.975262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.975239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ms892"]
Jul 28 01:26:37.975371 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.975356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:37.978668 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.978653    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.978751 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.978722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 01:26:37.978805 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.978767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9vjhm\""
Jul 28 01:26:37.978890 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.978875    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 01:26:37.979161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.979144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.984400 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.984383    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 01:26:37.999418 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.999397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:37.999497 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.999402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"]
Jul 28 01:26:37.999497 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.999449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-6k54l"]
Jul 28 01:26:37.999497 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:37.999466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ms892"]
Jul 28 01:26:38.002011 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.001995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bwtm6\""
Jul 28 01:26:38.002223 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.002208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:38.002295 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.002238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 01:26:38.002798 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.002780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:38.002876 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.002854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 01:26:38.008678 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.008661    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 01:26:38.033658 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.033635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c6360a-abac-4470-9a52-1eb23a783688-config\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.033738 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.033695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnfq\" (UniqueName: \"kubernetes.io/projected/69c6360a-abac-4470-9a52-1eb23a783688-kube-api-access-djnfq\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.033738 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.033721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm8qq\" (UniqueName: \"kubernetes.io/projected/185e9e72-f552-4f09-aebc-51714ec10f67-kube-api-access-nm8qq\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.033807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.033746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/185e9e72-f552-4f09-aebc-51714ec10f67-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.033807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.033761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.033896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.033809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c6360a-abac-4470-9a52-1eb23a783688-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.134156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-trusted-ca\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.134156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-serving-cert\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.134156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c6360a-abac-4470-9a52-1eb23a783688-config\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.134344 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.134383 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnfq\" (UniqueName: \"kubernetes.io/projected/69c6360a-abac-4470-9a52-1eb23a783688-kube-api-access-djnfq\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.134383 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8qq\" (UniqueName: \"kubernetes.io/projected/185e9e72-f552-4f09-aebc-51714ec10f67-kube-api-access-nm8qq\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.134473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-serving-cert\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.134473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gk6c\" (UniqueName: \"kubernetes.io/projected/ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b-kube-api-access-5gk6c\") pod \"network-check-source-7d88cb55d8-pzk4p\" (UID: \"ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"
Jul 28 01:26:38.134573 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/185e9e72-f552-4f09-aebc-51714ec10f67-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.134573 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.134573 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-tmp\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.134573 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-snapshots\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.134761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdsg\" (UniqueName: \"kubernetes.io/projected/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-kube-api-access-stdsg\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.134761 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:38.134620    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:38.134761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c6360a-abac-4470-9a52-1eb23a783688-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.134761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkvp\" (UniqueName: \"kubernetes.io/projected/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-kube-api-access-wmkvp\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.134761 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:38.134735    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls podName:185e9e72-f552-4f09-aebc-51714ec10f67 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:38.634686926 +0000 UTC m=+49.469434455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9vgj6" (UID: "185e9e72-f552-4f09-aebc-51714ec10f67") : secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:38.134968 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/69c6360a-abac-4470-9a52-1eb23a783688-config\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.134968 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-service-ca-bundle\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.134968 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.134806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-config\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.135177 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.135160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/185e9e72-f552-4f09-aebc-51714ec10f67-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.138318 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.138300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69c6360a-abac-4470-9a52-1eb23a783688-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.142547 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.142516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnfq\" (UniqueName: \"kubernetes.io/projected/69c6360a-abac-4470-9a52-1eb23a783688-kube-api-access-djnfq\") pod \"service-ca-operator-5d6c75f84f-lrl7s\" (UID: \"69c6360a-abac-4470-9a52-1eb23a783688\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.142994 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.142980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8qq\" (UniqueName: \"kubernetes.io/projected/185e9e72-f552-4f09-aebc-51714ec10f67-kube-api-access-nm8qq\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.158771 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.158751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"
Jul 28 01:26:38.235299 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-serving-cert\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.235429 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gk6c\" (UniqueName: \"kubernetes.io/projected/ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b-kube-api-access-5gk6c\") pod \"network-check-source-7d88cb55d8-pzk4p\" (UID: \"ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"
Jul 28 01:26:38.235429 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-tmp\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.235429 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-snapshots\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.235429 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdsg\" (UniqueName: \"kubernetes.io/projected/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-kube-api-access-stdsg\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.235837 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkvp\" (UniqueName: \"kubernetes.io/projected/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-kube-api-access-wmkvp\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.235970 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-tmp\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.235970 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-service-ca-bundle\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.235970 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-config\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.235970 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-trusted-ca\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.236203 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.235988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-serving-cert\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.236203 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.236027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.236315 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.236259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-snapshots\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.236950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.236831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-config\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.237104 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.237002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.237312 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.237251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-service-ca-bundle\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.237312 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.237298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-trusted-ca\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.237994 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.237969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-serving-cert\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.239177 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.239136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-serving-cert\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.244389 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.244358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkvp\" (UniqueName: \"kubernetes.io/projected/d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa-kube-api-access-wmkvp\") pod \"console-operator-584dcb6b54-ms892\" (UID: \"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.244495 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.244466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdsg\" (UniqueName: \"kubernetes.io/projected/6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3-kube-api-access-stdsg\") pod \"insights-operator-5885fdb759-6k54l\" (UID: \"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3\") " pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.244904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.244851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gk6c\" (UniqueName: \"kubernetes.io/projected/ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b-kube-api-access-5gk6c\") pod \"network-check-source-7d88cb55d8-pzk4p\" (UID: \"ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"
Jul 28 01:26:38.258262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.258090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"
Jul 28 01:26:38.284289 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.284269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-6k54l"
Jul 28 01:26:38.301560 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.301540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s"]
Jul 28 01:26:38.304169 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:38.304146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c6360a_abac_4470_9a52_1eb23a783688.slice/crio-4055ee160a64285cf8764c84208ceb7f3aa970d25da2c31582617c93cadb592c WatchSource:0}: Error finding container 4055ee160a64285cf8764c84208ceb7f3aa970d25da2c31582617c93cadb592c: Status 404 returned error can't find the container with id 4055ee160a64285cf8764c84208ceb7f3aa970d25da2c31582617c93cadb592c
Jul 28 01:26:38.307438 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.307420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:38.442700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.442634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p"]
Jul 28 01:26:38.469683 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.469649    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-6k54l"]
Jul 28 01:26:38.474353 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:38.474323    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7b9cf5_039e_4b0d_a0a8_e3b711f72cc3.slice/crio-add4ea15fd4ec0bd4ee8948e8cd6bb73643c66f96f2e761135cf50beebe07a40 WatchSource:0}: Error finding container add4ea15fd4ec0bd4ee8948e8cd6bb73643c66f96f2e761135cf50beebe07a40: Status 404 returned error can't find the container with id add4ea15fd4ec0bd4ee8948e8cd6bb73643c66f96f2e761135cf50beebe07a40
Jul 28 01:26:38.491827 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.491802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ms892"]
Jul 28 01:26:38.494460 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:38.494439    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b4a55f_79d1_43d3_92a6_aa7c5f3606fa.slice/crio-382525ede0f096aa25e64d465e2481ee5277123067a066e5e04df73fb0bcebdc WatchSource:0}: Error finding container 382525ede0f096aa25e64d465e2481ee5277123067a066e5e04df73fb0bcebdc: Status 404 returned error can't find the container with id 382525ede0f096aa25e64d465e2481ee5277123067a066e5e04df73fb0bcebdc
Jul 28 01:26:38.638171 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.638142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:38.638297 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:38.638254    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:38.638345 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:38.638306    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls podName:185e9e72-f552-4f09-aebc-51714ec10f67 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:39.638292357 +0000 UTC m=+50.473039880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9vgj6" (UID: "185e9e72-f552-4f09-aebc-51714ec10f67") : secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:38.969158 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.969123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" event={"ID":"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa","Type":"ContainerStarted","Data":"382525ede0f096aa25e64d465e2481ee5277123067a066e5e04df73fb0bcebdc"}
Jul 28 01:26:38.970198 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.970175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p" event={"ID":"ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b","Type":"ContainerStarted","Data":"36e53a0f83e18a19677db70b955820e8666b343370a3f2b8d46793f4cc903d76"}
Jul 28 01:26:38.971209 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.971188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6k54l" event={"ID":"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3","Type":"ContainerStarted","Data":"add4ea15fd4ec0bd4ee8948e8cd6bb73643c66f96f2e761135cf50beebe07a40"}
Jul 28 01:26:38.972257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:38.972238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s" event={"ID":"69c6360a-abac-4470-9a52-1eb23a783688","Type":"ContainerStarted","Data":"4055ee160a64285cf8764c84208ceb7f3aa970d25da2c31582617c93cadb592c"}
Jul 28 01:26:39.646747 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:39.646610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:39.647002 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:39.646757    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:39.647002 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:39.646813    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls podName:185e9e72-f552-4f09-aebc-51714ec10f67 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:41.646798421 +0000 UTC m=+52.481545949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9vgj6" (UID: "185e9e72-f552-4f09-aebc-51714ec10f67") : secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:41.460262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:41.460224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:41.460744 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:41.460362    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:41.460744 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:41.460430    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls podName:b6df3021-0059-41bc-be13-3067ea24b062 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:57.460412177 +0000 UTC m=+68.295159703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls") pod "dns-default-jpzzq" (UID: "b6df3021-0059-41bc-be13-3067ea24b062") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:41.661166 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:41.661123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:41.661339 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:41.661266    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:41.661339 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:41.661334    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls podName:185e9e72-f552-4f09-aebc-51714ec10f67 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.661314966 +0000 UTC m=+56.496062495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9vgj6" (UID: "185e9e72-f552-4f09-aebc-51714ec10f67") : secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:43.987984 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.987955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/0.log"
Jul 28 01:26:43.988493 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.988000    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa" containerID="2c032994d24bec0681f4b5c74e4a71544d9ed1197a0aff28b02c3ab7d23f7ba8" exitCode=255
Jul 28 01:26:43.988493 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.988081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" event={"ID":"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa","Type":"ContainerDied","Data":"2c032994d24bec0681f4b5c74e4a71544d9ed1197a0aff28b02c3ab7d23f7ba8"}
Jul 28 01:26:43.988493 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.988340    2577 scope.go:117] "RemoveContainer" containerID="2c032994d24bec0681f4b5c74e4a71544d9ed1197a0aff28b02c3ab7d23f7ba8"
Jul 28 01:26:43.990167 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.990135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p" event={"ID":"ef4cc8ff-fc5b-4bc2-95f1-a04f1fe54b0b","Type":"ContainerStarted","Data":"449990eda47b402c12a0238dc75ac37607cea5ba34748bfc51fef2a05f646995"}
Jul 28 01:26:43.992100 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.992073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6k54l" event={"ID":"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3","Type":"ContainerStarted","Data":"1531615f25edba0338067d2a314dc7c05c4ed529d0c34bc349a9029530a36a10"}
Jul 28 01:26:43.993600 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:43.993573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s" event={"ID":"69c6360a-abac-4470-9a52-1eb23a783688","Type":"ContainerStarted","Data":"b5ba7d9522c860f31b56ee54dec20b1b0167a39cf6cef7d74ade2c50f6716418"}
Jul 28 01:26:44.019170 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:44.019119    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pzk4p" podStartSLOduration=1.8692148899999999 podStartE2EDuration="7.019102129s" totalImagesPullingTime="5.149887239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:44.018353949 +0000 UTC m=+54.853101495" watchObservedRunningTime="2026-07-28 01:26:44.019102129 +0000 UTC m=+54.853849674"
Jul 28 01:26:44.037170 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:44.037114    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s" podStartSLOduration=1.751777634 podStartE2EDuration="7.037095727s" totalImagesPullingTime="5.285318093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:44.036174949 +0000 UTC m=+54.870922565" watchObservedRunningTime="2026-07-28 01:26:44.037095727 +0000 UTC m=+54.871843277"
Jul 28 01:26:44.052850 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:44.052807    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-6k54l" podStartSLOduration=1.924669417 podStartE2EDuration="7.052792397s" totalImagesPullingTime="5.12812298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:44.051837599 +0000 UTC m=+54.886585147" watchObservedRunningTime="2026-07-28 01:26:44.052792397 +0000 UTC m=+54.887539942"
Jul 28 01:26:44.999438 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:44.999406    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:26:44.999965 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:44.999945    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/0.log"
Jul 28 01:26:45.000033 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:44.999990    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa" containerID="086cab9f3180d6d2410d167c95857079766be9607a7bf9436016d9b8bb8b1ce0" exitCode=255
Jul 28 01:26:45.000129 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.000102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" event={"ID":"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa","Type":"ContainerDied","Data":"086cab9f3180d6d2410d167c95857079766be9607a7bf9436016d9b8bb8b1ce0"}
Jul 28 01:26:45.000207 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.000160    2577 scope.go:117] "RemoveContainer" containerID="2c032994d24bec0681f4b5c74e4a71544d9ed1197a0aff28b02c3ab7d23f7ba8"
Jul 28 01:26:45.000469 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.000355    2577 scope.go:117] "RemoveContainer" containerID="086cab9f3180d6d2410d167c95857079766be9607a7bf9436016d9b8bb8b1ce0"
Jul 28 01:26:45.000589 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.000548    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ms892_openshift-console-operator(d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" podUID="d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa"
Jul 28 01:26:45.495004 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.494972    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h5pr8"]
Jul 28 01:26:45.525527 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.525504    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h5pr8"]
Jul 28 01:26:45.525664 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.525623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.528754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.528491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:45.528754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.528568    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gqjq9\""
Jul 28 01:26:45.528754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.528491    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:45.591098 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.591069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bbxs\" (UniqueName: \"kubernetes.io/projected/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-api-access-9bbxs\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.591231 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.591137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-crio-socket\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.591231 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.591170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-data-volume\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.591231 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.591207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.591332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.591232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.691750 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.691896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bbxs\" (UniqueName: \"kubernetes.io/projected/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-api-access-9bbxs\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.691896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-crio-socket\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.691896 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.691839    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:45.691896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-data-volume\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.691925    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.191905084 +0000 UTC m=+57.026652628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.691964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-crio-socket\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.691990    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.692036    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls podName:185e9e72-f552-4f09-aebc-51714ec10f67 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.692028051 +0000 UTC m=+64.526775575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9vgj6" (UID: "185e9e72-f552-4f09-aebc-51714ec10f67") : secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:45.692096 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.692090    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:45.692340 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:45.692115    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.19210814 +0000 UTC m=+57.026855664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:45.692340 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.692298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-data-volume\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:45.702408 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:45.702382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bbxs\" (UniqueName: \"kubernetes.io/projected/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-api-access-9bbxs\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:46.005644 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:46.005609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:26:46.006154 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:46.006039    2577 scope.go:117] "RemoveContainer" containerID="086cab9f3180d6d2410d167c95857079766be9607a7bf9436016d9b8bb8b1ce0"
Jul 28 01:26:46.006254 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:46.006224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ms892_openshift-console-operator(d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" podUID="d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa"
Jul 28 01:26:46.197088 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:46.197057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:46.197230 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:46.197103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:46.197230 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:46.197194    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:46.197315 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:46.197231    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:46.197315 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:46.197269    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.197250522 +0000 UTC m=+58.031998064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:46.197315 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:46.197289    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.19727535 +0000 UTC m=+58.032022873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.105839 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.105806    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-6q7mp"]
Jul 28 01:26:47.126568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.126545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.132125 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.132102    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 01:26:47.132243 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.132160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 01:26:47.132693 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.132589    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 01:26:47.133117 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.133099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k27x7\""
Jul 28 01:26:47.133194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.133135    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 01:26:47.136565 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.136546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-6q7mp"]
Jul 28 01:26:47.206123 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.206091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3c3b78b0-b5af-49de-ab56-4ba100da0b77-signing-cabundle\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.206271 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.206147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:47.206271 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.206172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3c3b78b0-b5af-49de-ab56-4ba100da0b77-signing-key\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.206271 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.206224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:47.206371 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:47.206271    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.206371 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.206303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jbw\" (UniqueName: \"kubernetes.io/projected/3c3b78b0-b5af-49de-ab56-4ba100da0b77-kube-api-access-64jbw\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.206371 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:47.206332    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.206371 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:47.206335    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.206317503 +0000 UTC m=+60.041065050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.206493 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:47.206387    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.206374061 +0000 UTC m=+60.041121589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.281570 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.281549    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-74w4r_6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67/dns-node-resolver/0.log"
Jul 28 01:26:47.307202 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.307171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jbw\" (UniqueName: \"kubernetes.io/projected/3c3b78b0-b5af-49de-ab56-4ba100da0b77-kube-api-access-64jbw\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.307342 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.307228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3c3b78b0-b5af-49de-ab56-4ba100da0b77-signing-cabundle\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.307342 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.307287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3c3b78b0-b5af-49de-ab56-4ba100da0b77-signing-key\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.308435 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.308410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3c3b78b0-b5af-49de-ab56-4ba100da0b77-signing-cabundle\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.309674 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.309652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3c3b78b0-b5af-49de-ab56-4ba100da0b77-signing-key\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.320522 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.320502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jbw\" (UniqueName: \"kubernetes.io/projected/3c3b78b0-b5af-49de-ab56-4ba100da0b77-kube-api-access-64jbw\") pod \"service-ca-658d7b7c78-6q7mp\" (UID: \"3c3b78b0-b5af-49de-ab56-4ba100da0b77\") " pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.435340 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.435244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp"
Jul 28 01:26:47.559300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:47.559269    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-6q7mp"]
Jul 28 01:26:47.562275 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:47.562246    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3b78b0_b5af_49de_ab56_4ba100da0b77.slice/crio-43747066c3f25b1bd3fc4bc4435b5113dbe18f20692baf435cdf75a1d711e07a WatchSource:0}: Error finding container 43747066c3f25b1bd3fc4bc4435b5113dbe18f20692baf435cdf75a1d711e07a: Status 404 returned error can't find the container with id 43747066c3f25b1bd3fc4bc4435b5113dbe18f20692baf435cdf75a1d711e07a
Jul 28 01:26:48.012265 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.012230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp" event={"ID":"3c3b78b0-b5af-49de-ab56-4ba100da0b77","Type":"ContainerStarted","Data":"7f7352201046419184c732fdf1dc6d8a996043693ecf09bd7a21f6c0052ed7f0"}
Jul 28 01:26:48.012434 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.012276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp" event={"ID":"3c3b78b0-b5af-49de-ab56-4ba100da0b77","Type":"ContainerStarted","Data":"43747066c3f25b1bd3fc4bc4435b5113dbe18f20692baf435cdf75a1d711e07a"}
Jul 28 01:26:48.050796 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.050741    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-6q7mp" podStartSLOduration=1.050724424 podStartE2EDuration="1.050724424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:48.050320489 +0000 UTC m=+58.885068034" watchObservedRunningTime="2026-07-28 01:26:48.050724424 +0000 UTC m=+58.885471969"
Jul 28 01:26:48.282135 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.282067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2thtx_f0e34270-2b92-4098-93e6-028f9fe3f1ee/node-ca/0.log"
Jul 28 01:26:48.308457 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.308426    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:48.308620 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.308487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:26:48.308894 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.308855    2577 scope.go:117] "RemoveContainer" containerID="086cab9f3180d6d2410d167c95857079766be9607a7bf9436016d9b8bb8b1ce0"
Jul 28 01:26:48.309078 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:48.309057    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ms892_openshift-console-operator(d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" podUID="d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa"
Jul 28 01:26:48.886107 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.886056    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7v6r_764abbf1-6c23-4e88-adca-d80b035509ce/serve-healthcheck-canary/0.log"
Jul 28 01:26:48.916831 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:48.916805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccfz9"
Jul 28 01:26:49.224081 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:49.224050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:49.224264 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:49.224094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:49.224264 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:49.224186    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.224264 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:49.224215    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:49.224264 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:49.224246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.224231284 +0000 UTC m=+64.058978808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.224422 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:49.224276    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.224257539 +0000 UTC m=+64.059005064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.256880 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:53.256830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:53.257237 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:53.256890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:26:53.257237 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:53.256969    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:53.257237 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:53.257050    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.257237 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:53.257073    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:27:01.257057331 +0000 UTC m=+72.091804855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:53.257237 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:53.257107    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls podName:9f5c33a5-68fd-46cc-bd88-abb9aff18dfd nodeName:}" failed. No retries permitted until 2026-07-28 01:27:01.257096365 +0000 UTC m=+72.091843889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h5pr8" (UID: "9f5c33a5-68fd-46cc-bd88-abb9aff18dfd") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.760004 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:53.759974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:26:53.760155 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:53.760108    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:53.760210 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:53.760197    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls podName:185e9e72-f552-4f09-aebc-51714ec10f67 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:09.760179269 +0000 UTC m=+80.594926796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9vgj6" (UID: "185e9e72-f552-4f09-aebc-51714ec10f67") : secret "cluster-monitoring-operator-tls" not found
Jul 28 01:26:55.372098 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.372064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:26:55.372487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.372206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:55.376103 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.376080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:55.376205 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.376155    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:55.383119 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:55.383098    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 01:26:55.383207 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:26:55.383161    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs podName:fb8c2454-c3c5-4499-a2d8-1a5cfe353eba nodeName:}" failed. No retries permitted until 2026-07-28 01:27:59.383141328 +0000 UTC m=+130.217888852 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs") pod "network-metrics-daemon-qbb85" (UID: "fb8c2454-c3c5-4499-a2d8-1a5cfe353eba") : secret "metrics-daemon-secret" not found
Jul 28 01:26:55.385266 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.385244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/907860fa-083a-47ae-a4ad-dc7776e93632-original-pull-secret\") pod \"global-pull-secret-syncer-pqlq7\" (UID: \"907860fa-083a-47ae-a4ad-dc7776e93632\") " pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:55.400757 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.400739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pqlq7"
Jul 28 01:26:55.473268 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.473238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:55.476482 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.476448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp2f\" (UniqueName: \"kubernetes.io/projected/ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462-kube-api-access-zfp2f\") pod \"network-check-target-fnv6c\" (UID: \"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462\") " pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:55.535828 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.535801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pqlq7"]
Jul 28 01:26:55.539218 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:55.539178    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907860fa_083a_47ae_a4ad_dc7776e93632.slice/crio-f6d575bc8b24bab8cdb415b8e3936dbcee8c46cd3df37557def0d3de4b1ef4be WatchSource:0}: Error finding container f6d575bc8b24bab8cdb415b8e3936dbcee8c46cd3df37557def0d3de4b1ef4be: Status 404 returned error can't find the container with id f6d575bc8b24bab8cdb415b8e3936dbcee8c46cd3df37557def0d3de4b1ef4be
Jul 28 01:26:55.708573 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.708504    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4mq45\""
Jul 28 01:26:55.716506 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.716490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:55.851902 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:55.851855    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fnv6c"]
Jul 28 01:26:55.854249 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:55.854225    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc7ba6f_b4e9_4b93_939a_3c9aa9b80462.slice/crio-cb7609694e7a6802e72b532a7adcbc1d3fddfbe53c734aaa103e4dd0601ad6aa WatchSource:0}: Error finding container cb7609694e7a6802e72b532a7adcbc1d3fddfbe53c734aaa103e4dd0601ad6aa: Status 404 returned error can't find the container with id cb7609694e7a6802e72b532a7adcbc1d3fddfbe53c734aaa103e4dd0601ad6aa
Jul 28 01:26:56.038429 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:56.038392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fnv6c" event={"ID":"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462","Type":"ContainerStarted","Data":"22d4441e36017903f42427098a12eff2d0a7d62d8a331dc989e07f11adfb3758"}
Jul 28 01:26:56.038617 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:56.038477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fnv6c" event={"ID":"ebc7ba6f-b4e9-4b93-939a-3c9aa9b80462","Type":"ContainerStarted","Data":"cb7609694e7a6802e72b532a7adcbc1d3fddfbe53c734aaa103e4dd0601ad6aa"}
Jul 28 01:26:56.038617 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:56.038527    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:26:56.039906 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:56.039884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pqlq7" event={"ID":"907860fa-083a-47ae-a4ad-dc7776e93632","Type":"ContainerStarted","Data":"f6d575bc8b24bab8cdb415b8e3936dbcee8c46cd3df37557def0d3de4b1ef4be"}
Jul 28 01:26:56.067734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:56.067693    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fnv6c" podStartSLOduration=66.0676811 podStartE2EDuration="1m6.0676811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:56.067533677 +0000 UTC m=+66.902281238" watchObservedRunningTime="2026-07-28 01:26:56.0676811 +0000 UTC m=+66.902428646"
Jul 28 01:26:57.490004 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:57.489962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:57.492559 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:57.492532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6df3021-0059-41bc-be13-3067ea24b062-metrics-tls\") pod \"dns-default-jpzzq\" (UID: \"b6df3021-0059-41bc-be13-3067ea24b062\") " pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:57.747898 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:57.747808    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mh7n2\""
Jul 28 01:26:57.755365 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:57.755341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:26:57.908614 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:57.908586    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jpzzq"]
Jul 28 01:26:57.912642 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:26:57.912605    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6df3021_0059_41bc_be13_3067ea24b062.slice/crio-ca67ee0c224d7464efa27d2ec237824a504f42e33fd6bb7b75cee3e728a09b1c WatchSource:0}: Error finding container ca67ee0c224d7464efa27d2ec237824a504f42e33fd6bb7b75cee3e728a09b1c: Status 404 returned error can't find the container with id ca67ee0c224d7464efa27d2ec237824a504f42e33fd6bb7b75cee3e728a09b1c
Jul 28 01:26:58.050960 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:26:58.050883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpzzq" event={"ID":"b6df3021-0059-41bc-be13-3067ea24b062","Type":"ContainerStarted","Data":"ca67ee0c224d7464efa27d2ec237824a504f42e33fd6bb7b75cee3e728a09b1c"}
Jul 28 01:27:00.059547 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:00.059501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pqlq7" event={"ID":"907860fa-083a-47ae-a4ad-dc7776e93632","Type":"ContainerStarted","Data":"04809b430a0ea3c0a6b052b3d453c20f0e73de52b2467e263661e24385b42dc0"}
Jul 28 01:27:00.081154 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:00.081101    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pqlq7" podStartSLOduration=65.93110535 podStartE2EDuration="1m10.081084716s" totalImagesPullingTime="4.149979366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:00.081020531 +0000 UTC m=+70.915768078" watchObservedRunningTime="2026-07-28 01:27:00.081084716 +0000 UTC m=+70.915832264"
Jul 28 01:27:00.691690 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:00.691653    2577 scope.go:117] "RemoveContainer" containerID="086cab9f3180d6d2410d167c95857079766be9607a7bf9436016d9b8bb8b1ce0"
Jul 28 01:27:01.064048 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.064016    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:27:01.064418 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.064124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" event={"ID":"d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa","Type":"ContainerStarted","Data":"287f272ae02831e99f82d576ede6c5c1d9e4ee08cda09eef7877ce705c002c9e"}
Jul 28 01:27:01.064489 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.064414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:27:01.066060 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.066035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpzzq" event={"ID":"b6df3021-0059-41bc-be13-3067ea24b062","Type":"ContainerStarted","Data":"4daf03706415c072469f8b35ad6b9dd112956f0dcf8a7df03cf51a9d31d3863e"}
Jul 28 01:27:01.066166 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.066081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jpzzq" event={"ID":"b6df3021-0059-41bc-be13-3067ea24b062","Type":"ContainerStarted","Data":"38d2491173e2ea90b2138e75b04083cd3851642d9df14ffbe89361648306aae2"}
Jul 28 01:27:01.066166 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.066113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:27:01.083823 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.083781    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-ms892" podStartSLOduration=18.982116492 podStartE2EDuration="24.083769068s" totalImagesPullingTime="5.101652576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:01.082627567 +0000 UTC m=+71.917375114" watchObservedRunningTime="2026-07-28 01:27:01.083769068 +0000 UTC m=+71.918516614"
Jul 28 01:27:01.102066 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.102020    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jpzzq" podStartSLOduration=33.180227534 podStartE2EDuration="36.102003313s" totalImagesPullingTime="2.921775779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:01.101085941 +0000 UTC m=+71.935833488" watchObservedRunningTime="2026-07-28 01:27:01.102003313 +0000 UTC m=+71.936750857"
Jul 28 01:27:01.323660 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.323632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:27:01.323815 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.323736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:27:01.324134 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.324113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:27:01.325987 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.325967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f5c33a5-68fd-46cc-bd88-abb9aff18dfd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h5pr8\" (UID: \"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd\") " pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:27:01.440454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.440421    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gqjq9\""
Jul 28 01:27:01.440755 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.440729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-ms892"
Jul 28 01:27:01.448914 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.448892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h5pr8"
Jul 28 01:27:01.595015 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:01.594935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h5pr8"]
Jul 28 01:27:01.598098 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:01.598060    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5c33a5_68fd_46cc_bd88_abb9aff18dfd.slice/crio-634149c3c25d5f2c02d2a91d983c9df9dddf372af6dabca6fb91d582d3c5e75d WatchSource:0}: Error finding container 634149c3c25d5f2c02d2a91d983c9df9dddf372af6dabca6fb91d582d3c5e75d: Status 404 returned error can't find the container with id 634149c3c25d5f2c02d2a91d983c9df9dddf372af6dabca6fb91d582d3c5e75d
Jul 28 01:27:02.070954 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:02.070912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5pr8" event={"ID":"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd","Type":"ContainerStarted","Data":"634149c3c25d5f2c02d2a91d983c9df9dddf372af6dabca6fb91d582d3c5e75d"}
Jul 28 01:27:07.091193 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:07.091161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5pr8" event={"ID":"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd","Type":"ContainerStarted","Data":"f9e5e5039806da929dca0640dc0ae0f4094e9d68da90c9593f65996445a43999"}
Jul 28 01:27:09.654675 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.654644    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"]
Jul 28 01:27:09.660080 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.660060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.681582 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.681523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 01:27:09.681763 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.681524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 01:27:09.681763 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.681523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 01:27:09.696185 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.696161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cwxnf\""
Jul 28 01:27:09.696336 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.696281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 01:27:09.725961 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.725923    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"]
Jul 28 01:27:09.801139 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.801096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/123779c5-56af-410e-98d3-4ccd5264d62c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbdfd7698-4wp59\" (UID: \"123779c5-56af-410e-98d3-4ccd5264d62c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.801331 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.801144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqg2\" (UniqueName: \"kubernetes.io/projected/123779c5-56af-410e-98d3-4ccd5264d62c-kube-api-access-shqg2\") pod \"managed-serviceaccount-addon-agent-6cbdfd7698-4wp59\" (UID: \"123779c5-56af-410e-98d3-4ccd5264d62c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.801331 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.801234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:27:09.804237 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.804209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/185e9e72-f552-4f09-aebc-51714ec10f67-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9vgj6\" (UID: \"185e9e72-f552-4f09-aebc-51714ec10f67\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:27:09.902426 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.902383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/123779c5-56af-410e-98d3-4ccd5264d62c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbdfd7698-4wp59\" (UID: \"123779c5-56af-410e-98d3-4ccd5264d62c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.902426 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.902429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqg2\" (UniqueName: \"kubernetes.io/projected/123779c5-56af-410e-98d3-4ccd5264d62c-kube-api-access-shqg2\") pod \"managed-serviceaccount-addon-agent-6cbdfd7698-4wp59\" (UID: \"123779c5-56af-410e-98d3-4ccd5264d62c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.905281 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.905207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/123779c5-56af-410e-98d3-4ccd5264d62c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbdfd7698-4wp59\" (UID: \"123779c5-56af-410e-98d3-4ccd5264d62c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.914747 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.914716    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-659f8869dc-t2rzb"]
Jul 28 01:27:09.917901 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.917322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:09.919761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.919729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqg2\" (UniqueName: \"kubernetes.io/projected/123779c5-56af-410e-98d3-4ccd5264d62c-kube-api-access-shqg2\") pod \"managed-serviceaccount-addon-agent-6cbdfd7698-4wp59\" (UID: \"123779c5-56af-410e-98d3-4ccd5264d62c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:09.923582 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.923559    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 01:27:09.923696 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.923642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knzvx\""
Jul 28 01:27:09.923696 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.923663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 01:27:09.923805 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.923559    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 01:27:09.931464 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.931298    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 01:27:09.935437 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.935340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659f8869dc-t2rzb"]
Jul 28 01:27:09.973771 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.973735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-szd96\""
Jul 28 01:27:09.981797 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.981768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"
Jul 28 01:27:09.982910 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:09.982889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"
Jul 28 01:27:10.003565 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-registry-tls\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003710 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe286845-7f5c-4f55-a96a-21cc7efc794f-registry-certificates\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003710 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe286845-7f5c-4f55-a96a-21cc7efc794f-installation-pull-secrets\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003798 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe286845-7f5c-4f55-a96a-21cc7efc794f-trusted-ca\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003798 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe286845-7f5c-4f55-a96a-21cc7efc794f-ca-trust-extracted\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003915 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-bound-sa-token\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003951 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml58p\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-kube-api-access-ml58p\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.003999 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.003946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe286845-7f5c-4f55-a96a-21cc7efc794f-image-registry-private-configuration\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104285 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-registry-tls\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104421 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe286845-7f5c-4f55-a96a-21cc7efc794f-registry-certificates\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104485 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe286845-7f5c-4f55-a96a-21cc7efc794f-installation-pull-secrets\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe286845-7f5c-4f55-a96a-21cc7efc794f-trusted-ca\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe286845-7f5c-4f55-a96a-21cc7efc794f-ca-trust-extracted\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104638 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-bound-sa-token\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104638 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml58p\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-kube-api-access-ml58p\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.104638 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.104626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe286845-7f5c-4f55-a96a-21cc7efc794f-image-registry-private-configuration\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.105616 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.105307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe286845-7f5c-4f55-a96a-21cc7efc794f-ca-trust-extracted\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.105616 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.105525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5pr8" event={"ID":"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd","Type":"ContainerStarted","Data":"dab24f6eab3467a60a4fc0e46f31bdb4c22d35ce07dca3af16fe03b949e95085"}
Jul 28 01:27:10.106472 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.106412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe286845-7f5c-4f55-a96a-21cc7efc794f-registry-certificates\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.106887 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.106843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe286845-7f5c-4f55-a96a-21cc7efc794f-trusted-ca\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.107915 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.107892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-registry-tls\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.109154 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.109131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe286845-7f5c-4f55-a96a-21cc7efc794f-installation-pull-secrets\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.109479 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.109446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe286845-7f5c-4f55-a96a-21cc7efc794f-image-registry-private-configuration\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.118558 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.118497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-bound-sa-token\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.120063 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.120035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml58p\" (UniqueName: \"kubernetes.io/projected/fe286845-7f5c-4f55-a96a-21cc7efc794f-kube-api-access-ml58p\") pod \"image-registry-659f8869dc-t2rzb\" (UID: \"fe286845-7f5c-4f55-a96a-21cc7efc794f\") " pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.140845 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.140810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59"]
Jul 28 01:27:10.144012 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:10.143987    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod123779c5_56af_410e_98d3_4ccd5264d62c.slice/crio-7968db8b2e913edf0f794e1bbeb4f0dc1b9984266bdd93e99d67457e4cd069f8 WatchSource:0}: Error finding container 7968db8b2e913edf0f794e1bbeb4f0dc1b9984266bdd93e99d67457e4cd069f8: Status 404 returned error can't find the container with id 7968db8b2e913edf0f794e1bbeb4f0dc1b9984266bdd93e99d67457e4cd069f8
Jul 28 01:27:10.155381 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.155324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6"]
Jul 28 01:27:10.158441 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:10.158417    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185e9e72_f552_4f09_aebc_51714ec10f67.slice/crio-eb2d4fe714d5f174b9721c9eb6b03cefeea763be49739bbf9e8baee2517f6ce1 WatchSource:0}: Error finding container eb2d4fe714d5f174b9721c9eb6b03cefeea763be49739bbf9e8baee2517f6ce1: Status 404 returned error can't find the container with id eb2d4fe714d5f174b9721c9eb6b03cefeea763be49739bbf9e8baee2517f6ce1
Jul 28 01:27:10.239227 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.239196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:10.367427 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:10.367396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659f8869dc-t2rzb"]
Jul 28 01:27:10.370422 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:10.370390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe286845_7f5c_4f55_a96a_21cc7efc794f.slice/crio-f09cf63ffeb7f7b235c7a696d4499139f343d5ede95dbc0c23f4519f7b9a6875 WatchSource:0}: Error finding container f09cf63ffeb7f7b235c7a696d4499139f343d5ede95dbc0c23f4519f7b9a6875: Status 404 returned error can't find the container with id f09cf63ffeb7f7b235c7a696d4499139f343d5ede95dbc0c23f4519f7b9a6875
Jul 28 01:27:11.073643 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.073524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jpzzq"
Jul 28 01:27:11.114792 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.113939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb" event={"ID":"fe286845-7f5c-4f55-a96a-21cc7efc794f","Type":"ContainerStarted","Data":"1533354be28292de2b109e4a08c7c9b0b0511c4c62a59a53543f00f433f745bb"}
Jul 28 01:27:11.114792 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.114004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb" event={"ID":"fe286845-7f5c-4f55-a96a-21cc7efc794f","Type":"ContainerStarted","Data":"f09cf63ffeb7f7b235c7a696d4499139f343d5ede95dbc0c23f4519f7b9a6875"}
Jul 28 01:27:11.114792 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.114734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:11.118431 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.118383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59" event={"ID":"123779c5-56af-410e-98d3-4ccd5264d62c","Type":"ContainerStarted","Data":"7968db8b2e913edf0f794e1bbeb4f0dc1b9984266bdd93e99d67457e4cd069f8"}
Jul 28 01:27:11.120461 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.120436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6" event={"ID":"185e9e72-f552-4f09-aebc-51714ec10f67","Type":"ContainerStarted","Data":"eb2d4fe714d5f174b9721c9eb6b03cefeea763be49739bbf9e8baee2517f6ce1"}
Jul 28 01:27:11.134293 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:11.134232    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb" podStartSLOduration=2.134211943 podStartE2EDuration="2.134211943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.133933884 +0000 UTC m=+81.968681433" watchObservedRunningTime="2026-07-28 01:27:11.134211943 +0000 UTC m=+81.968959468"
Jul 28 01:27:16.143495 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:16.143454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59" event={"ID":"123779c5-56af-410e-98d3-4ccd5264d62c","Type":"ContainerStarted","Data":"c22e059851131dd3cbe54ef23d926c5cbae2ed5719292e2f5e1281753c7fbf9f"}
Jul 28 01:27:16.145105 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:16.145080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6" event={"ID":"185e9e72-f552-4f09-aebc-51714ec10f67","Type":"ContainerStarted","Data":"53de9acdebb760bd17f83539d9752587ec3c91f0d7a87887ea7590b0c154b78b"}
Jul 28 01:27:16.147221 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:16.147195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h5pr8" event={"ID":"9f5c33a5-68fd-46cc-bd88-abb9aff18dfd","Type":"ContainerStarted","Data":"7655e605e4478260c463370dfc2d0badd8e47882a3abc52d3d064ca3c7a3b14d"}
Jul 28 01:27:16.160162 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:16.160115    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbdfd7698-4wp59" podStartSLOduration=1.8856454459999998 podStartE2EDuration="7.160096697s" totalImagesPullingTime="5.274451251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.159007746 +0000 UTC m=+86.993755300" watchObservedRunningTime="2026-07-28 01:27:16.160096697 +0000 UTC m=+86.994844250"
Jul 28 01:27:16.176771 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:16.176723    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9vgj6" podStartSLOduration=33.926203144 podStartE2EDuration="39.176706662s" totalImagesPullingTime="5.250503518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.175407975 +0000 UTC m=+87.010155524" watchObservedRunningTime="2026-07-28 01:27:16.176706662 +0000 UTC m=+87.011454208"
Jul 28 01:27:16.195169 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:16.195122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h5pr8" podStartSLOduration=17.503769612 podStartE2EDuration="31.195106976s" totalImagesPullingTime="13.691337364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.193809141 +0000 UTC m=+87.028556685" watchObservedRunningTime="2026-07-28 01:27:16.195106976 +0000 UTC m=+87.029854523"
Jul 28 01:27:27.045389 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:27.045354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fnv6c"
Jul 28 01:27:33.135610 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:33.135581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-659f8869dc-t2rzb"
Jul 28 01:27:34.382779 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.382743    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"]
Jul 28 01:27:34.386543 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.386522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.390402 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.390381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cdc6q\""
Jul 28 01:27:34.390814 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.390528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:27:34.391056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.390594    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 01:27:34.391227 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.390630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 01:27:34.405229 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.405209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"]
Jul 28 01:27:34.420176 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.420151    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ks6hc"]
Jul 28 01:27:34.422216 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.422198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.425184 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.425166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vt9th\""
Jul 28 01:27:34.425596 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.425576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:27:34.425856 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.425818    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:27:34.426087 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.426070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:27:34.487196 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-textfile\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487196 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-tls\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee46962-4618-4584-8574-622956fbf5e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-sys\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q8c\" (UniqueName: \"kubernetes.io/projected/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-kube-api-access-c8q8c\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-wtmp\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-metrics-client-ca\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487404 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-root\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.487744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cee46962-4618-4584-8574-622956fbf5e0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.487744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cee46962-4618-4584-8574-622956fbf5e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.487744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.487552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5m7f\" (UniqueName: \"kubernetes.io/projected/cee46962-4618-4584-8574-622956fbf5e0-kube-api-access-b5m7f\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.588247 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588247 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cee46962-4618-4584-8574-622956fbf5e0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cee46962-4618-4584-8574-622956fbf5e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5m7f\" (UniqueName: \"kubernetes.io/projected/cee46962-4618-4584-8574-622956fbf5e0-kube-api-access-b5m7f\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-textfile\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-tls\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee46962-4618-4584-8574-622956fbf5e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-sys\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q8c\" (UniqueName: \"kubernetes.io/projected/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-kube-api-access-c8q8c\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-wtmp\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-metrics-client-ca\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-root\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:27:34.588643    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-sys\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:27:34.588713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-tls podName:ec575a4e-f6d7-495f-9b5b-cf38ac5107e4 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:35.088695089 +0000 UTC m=+105.923442617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-tls") pod "node-exporter-ks6hc" (UID: "ec575a4e-f6d7-495f-9b5b-cf38ac5107e4") : secret "node-exporter-tls" not found
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-textfile\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-wtmp\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.588812 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.588647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-root\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.589350 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.589052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cee46962-4618-4584-8574-622956fbf5e0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.589350 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.589225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.589350 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.589327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-metrics-client-ca\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.591247 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.591227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cee46962-4618-4584-8574-622956fbf5e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.591505 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.591485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.591547 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.591524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cee46962-4618-4584-8574-622956fbf5e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.600667 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.600642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q8c\" (UniqueName: \"kubernetes.io/projected/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-kube-api-access-c8q8c\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:34.601058 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.601037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5m7f\" (UniqueName: \"kubernetes.io/projected/cee46962-4618-4584-8574-622956fbf5e0-kube-api-access-b5m7f\") pod \"openshift-state-metrics-b8f49f5c6-47wrj\" (UID: \"cee46962-4618-4584-8574-622956fbf5e0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.697995 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.697933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"
Jul 28 01:27:34.836490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:34.836458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj"]
Jul 28 01:27:34.839676 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:34.839648    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee46962_4618_4584_8574_622956fbf5e0.slice/crio-b3f82b50c2b50d479159b9a256a146a71aa098a05b3abce186ced0fa48e2babd WatchSource:0}: Error finding container b3f82b50c2b50d479159b9a256a146a71aa098a05b3abce186ced0fa48e2babd: Status 404 returned error can't find the container with id b3f82b50c2b50d479159b9a256a146a71aa098a05b3abce186ced0fa48e2babd
Jul 28 01:27:35.093337 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.093290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-tls\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:35.095494 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.095473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec575a4e-f6d7-495f-9b5b-cf38ac5107e4-node-exporter-tls\") pod \"node-exporter-ks6hc\" (UID: \"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4\") " pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:35.212289 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.212193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj" event={"ID":"cee46962-4618-4584-8574-622956fbf5e0","Type":"ContainerStarted","Data":"f13fc531921e256e2251fdc748dc0cffca5d66f120ee20377aa78a8057797661"}
Jul 28 01:27:35.212289 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.212278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj" event={"ID":"cee46962-4618-4584-8574-622956fbf5e0","Type":"ContainerStarted","Data":"3eef0badbbbcf177dbf1a9b4c4d02166b651ff56503474730ef2a1b9ece65747"}
Jul 28 01:27:35.212453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.212308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj" event={"ID":"cee46962-4618-4584-8574-622956fbf5e0","Type":"ContainerStarted","Data":"b3f82b50c2b50d479159b9a256a146a71aa098a05b3abce186ced0fa48e2babd"}
Jul 28 01:27:35.332677 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.332648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ks6hc"
Jul 28 01:27:35.344733 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:35.344708    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec575a4e_f6d7_495f_9b5b_cf38ac5107e4.slice/crio-badfb15e71921d0cd37c8d472c7add92176dc6324cc4ed375491fde62c1bf8de WatchSource:0}: Error finding container badfb15e71921d0cd37c8d472c7add92176dc6324cc4ed375491fde62c1bf8de: Status 404 returned error can't find the container with id badfb15e71921d0cd37c8d472c7add92176dc6324cc4ed375491fde62c1bf8de
Jul 28 01:27:35.497300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.497271    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:27:35.500833 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.500797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.503546 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.503523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 01:27:35.503688 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.503673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 01:27:35.503754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.503681    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 01:27:35.503754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.503691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dmmlf\""
Jul 28 01:27:35.503889 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.503815    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 01:27:35.503889 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.503821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 01:27:35.504093 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.504075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 01:27:35.504192 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.504076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 01:27:35.504192 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.504144    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 01:27:35.504192 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.504168    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 01:27:35.519021 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.518998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:27:35.597291 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597434 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0ba98e6-e092-4907-b35a-9c427cffaaaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597434 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0ba98e6-e092-4907-b35a-9c427cffaaaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597434 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597434 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0ba98e6-e092-4907-b35a-9c427cffaaaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0ba98e6-e092-4907-b35a-9c427cffaaaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-web-config\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0ba98e6-e092-4907-b35a-9c427cffaaaa-config-out\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lqg\" (UniqueName: \"kubernetes.io/projected/d0ba98e6-e092-4907-b35a-9c427cffaaaa-kube-api-access-h2lqg\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.597834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.598009 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.597836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0ba98e6-e092-4907-b35a-9c427cffaaaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0ba98e6-e092-4907-b35a-9c427cffaaaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.698813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0ba98e6-e092-4907-b35a-9c427cffaaaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0ba98e6-e092-4907-b35a-9c427cffaaaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-web-config\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0ba98e6-e092-4907-b35a-9c427cffaaaa-config-out\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.698962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.699006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lqg\" (UniqueName: \"kubernetes.io/projected/d0ba98e6-e092-4907-b35a-9c427cffaaaa-kube-api-access-h2lqg\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.699745 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.699721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0ba98e6-e092-4907-b35a-9c427cffaaaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.701074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.700667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0ba98e6-e092-4907-b35a-9c427cffaaaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.701074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.700937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0ba98e6-e092-4907-b35a-9c427cffaaaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.701720 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.701689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0ba98e6-e092-4907-b35a-9c427cffaaaa-config-out\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.702216 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.702190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.703245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.703222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.703347 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.703300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.704008 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.703770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.704008 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.703958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.704520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.704486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.705225 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.705202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0ba98e6-e092-4907-b35a-9c427cffaaaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.705334 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.705314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0ba98e6-e092-4907-b35a-9c427cffaaaa-web-config\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.708173 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.708154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lqg\" (UniqueName: \"kubernetes.io/projected/d0ba98e6-e092-4907-b35a-9c427cffaaaa-kube-api-access-h2lqg\") pod \"alertmanager-main-0\" (UID: \"d0ba98e6-e092-4907-b35a-9c427cffaaaa\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.812365 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.812289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:27:35.985709 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:35.985686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:27:35.987852 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:35.987828    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ba98e6_e092_4907_b35a_9c427cffaaaa.slice/crio-6ccc7c9cdf6413e688efd37990ca6fe716401c24f4b0b4ef102fdf7154e291ae WatchSource:0}: Error finding container 6ccc7c9cdf6413e688efd37990ca6fe716401c24f4b0b4ef102fdf7154e291ae: Status 404 returned error can't find the container with id 6ccc7c9cdf6413e688efd37990ca6fe716401c24f4b0b4ef102fdf7154e291ae
Jul 28 01:27:36.219100 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:36.219064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj" event={"ID":"cee46962-4618-4584-8574-622956fbf5e0","Type":"ContainerStarted","Data":"2a9d7e894e63e1ec4ac0710b34b415101393162afb48840c4c409153d92abb7d"}
Jul 28 01:27:36.220433 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:36.220407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"6ccc7c9cdf6413e688efd37990ca6fe716401c24f4b0b4ef102fdf7154e291ae"}
Jul 28 01:27:36.226371 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:36.226326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks6hc" event={"ID":"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4","Type":"ContainerStarted","Data":"badfb15e71921d0cd37c8d472c7add92176dc6324cc4ed375491fde62c1bf8de"}
Jul 28 01:27:36.237232 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:36.237194    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-47wrj" podStartSLOduration=1.33270748 podStartE2EDuration="2.237181935s" totalImagesPullingTime="904.474455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:36.23582833 +0000 UTC m=+107.070575877" watchObservedRunningTime="2026-07-28 01:27:36.237181935 +0000 UTC m=+107.071929482"
Jul 28 01:27:37.231394 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:37.231364    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0ba98e6-e092-4907-b35a-9c427cffaaaa" containerID="9d4b9c4fbb8fa35eb9a02227714564be241cbe8003bfffb07c01d73d90f474e0" exitCode=0
Jul 28 01:27:37.232061 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:37.231441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerDied","Data":"9d4b9c4fbb8fa35eb9a02227714564be241cbe8003bfffb07c01d73d90f474e0"}
Jul 28 01:27:37.233189 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:37.233167    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec575a4e-f6d7-495f-9b5b-cf38ac5107e4" containerID="025a9abf0da10d2a847bbdc02da4b93eeae54f3cb0643a752c1464490c5603d0" exitCode=0
Jul 28 01:27:37.233278 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:37.233255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks6hc" event={"ID":"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4","Type":"ContainerDied","Data":"025a9abf0da10d2a847bbdc02da4b93eeae54f3cb0643a752c1464490c5603d0"}
Jul 28 01:27:38.239496 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.239462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks6hc" event={"ID":"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4","Type":"ContainerStarted","Data":"f93849c3859d942e0233fefb061ab30bbeeb842ae6b9dc04980d5f9c942b9779"}
Jul 28 01:27:38.239943 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.239522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ks6hc" event={"ID":"ec575a4e-f6d7-495f-9b5b-cf38ac5107e4","Type":"ContainerStarted","Data":"b355987d9126427a7b2902ffac3618442af6c19fc85019976f560beb6a871345"}
Jul 28 01:27:38.264284 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.264226    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ks6hc" podStartSLOduration=3.176866297 podStartE2EDuration="4.264208274s" totalImagesPullingTime="1.020283843s" totalInitContainerRuntime="67.058134ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:38.262888702 +0000 UTC m=+109.097636249" watchObservedRunningTime="2026-07-28 01:27:38.264208274 +0000 UTC m=+109.098955822"
Jul 28 01:27:38.923825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.923789    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c5b67497c-gr5jc"]
Jul 28 01:27:38.931124 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.931106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:38.934066 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.933755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 01:27:38.934066 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.933920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dgm6v\""
Jul 28 01:27:38.934066 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.933967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-53eh03dvg2vhu\""
Jul 28 01:27:38.934330 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.934229    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 01:27:38.935881 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.934437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 01:27:38.935881 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.934486    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 01:27:38.938800 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:38.938778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c5b67497c-gr5jc"]
Jul 28 01:27:39.029811 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.029737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f263054-4032-44e7-9566-15acc9322453-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.029811 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.029788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-secret-metrics-server-tls\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.030053 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.029871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-secret-metrics-server-client-certs\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.030053 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.029933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8f263054-4032-44e7-9566-15acc9322453-metrics-server-audit-profiles\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.030053 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.029965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-client-ca-bundle\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.030053 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.029986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzl5k\" (UniqueName: \"kubernetes.io/projected/8f263054-4032-44e7-9566-15acc9322453-kube-api-access-gzl5k\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.030228 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.030111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8f263054-4032-44e7-9566-15acc9322453-audit-log\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.130928 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.130881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-secret-metrics-server-client-certs\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.131092 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.130945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8f263054-4032-44e7-9566-15acc9322453-metrics-server-audit-profiles\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.131092 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.130986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-client-ca-bundle\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.131092 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.131014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzl5k\" (UniqueName: \"kubernetes.io/projected/8f263054-4032-44e7-9566-15acc9322453-kube-api-access-gzl5k\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.131092 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.131082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8f263054-4032-44e7-9566-15acc9322453-audit-log\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.131303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.131114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f263054-4032-44e7-9566-15acc9322453-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.131303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.131158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-secret-metrics-server-tls\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.132453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.131725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8f263054-4032-44e7-9566-15acc9322453-audit-log\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.132453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.132071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8f263054-4032-44e7-9566-15acc9322453-metrics-server-audit-profiles\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.132453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.132412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f263054-4032-44e7-9566-15acc9322453-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.134410 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.134383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-secret-metrics-server-tls\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.134514 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.134466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-secret-metrics-server-client-certs\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.134832 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.134802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8f263054-4032-44e7-9566-15acc9322453-client-ca-bundle\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.163728 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.163621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzl5k\" (UniqueName: \"kubernetes.io/projected/8f263054-4032-44e7-9566-15acc9322453-kube-api-access-gzl5k\") pod \"metrics-server-c5b67497c-gr5jc\" (UID: \"8f263054-4032-44e7-9566-15acc9322453\") " pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.173497 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.173473    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"]
Jul 28 01:27:39.194627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.194604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.197205 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.197143    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 01:27:39.197523 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.197439    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2vxnv\""
Jul 28 01:27:39.198575 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.198501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"]
Jul 28 01:27:39.231568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.231548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.231755 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.231580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2md5k\" (UniqueName: \"kubernetes.io/projected/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-kube-api-access-2md5k\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.245002 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.244980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:39.247787 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.247763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"ed015fdf8821ddf612acadc5c7a5a229f1538b9accd05015adaa83a6f3df79f5"}
Jul 28 01:27:39.247913 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.247837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"e157d2246ad8c5a2144ab074c05299e245eb355c423c894a598271c072da4a7a"}
Jul 28 01:27:39.247913 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.247895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"094fb2738b81926b73b31f9cd14e35e8e4f631f8f095f367fc5b84a217548634"}
Jul 28 01:27:39.248039 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.247945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"d588f592c376ed00e1ce06570627e403b7a595deca5c003d2a15904dee521a47"}
Jul 28 01:27:39.248039 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.247989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"aeb986f9072e75044c5ede54277c45aad924251487df5e5fdb272d00ac291f74"}
Jul 28 01:27:39.332458 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.332389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.332458 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.332447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2md5k\" (UniqueName: \"kubernetes.io/projected/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-kube-api-access-2md5k\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.332695 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:27:39.332675    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 01:27:39.332753 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:27:39.332738    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-monitoring-plugin-cert podName:9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:39.832719025 +0000 UTC m=+110.667466555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-5bx4k" (UID: "9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84") : secret "monitoring-plugin-cert" not found
Jul 28 01:27:39.342642 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.342619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2md5k\" (UniqueName: \"kubernetes.io/projected/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-kube-api-access-2md5k\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.596628 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.596603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c5b67497c-gr5jc"]
Jul 28 01:27:39.600008 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:39.599986    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f263054_4032_44e7_9566_15acc9322453.slice/crio-5845d93c1e771a554fb7343f3e5930e60518c9a437bfdd6bf5b1eaf9a60e8637 WatchSource:0}: Error finding container 5845d93c1e771a554fb7343f3e5930e60518c9a437bfdd6bf5b1eaf9a60e8637: Status 404 returned error can't find the container with id 5845d93c1e771a554fb7343f3e5930e60518c9a437bfdd6bf5b1eaf9a60e8637
Jul 28 01:27:39.835926 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.835845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:39.838276 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:39.838218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-5bx4k\" (UID: \"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:40.107151 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:40.107068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:40.257250 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:40.257216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0ba98e6-e092-4907-b35a-9c427cffaaaa","Type":"ContainerStarted","Data":"a9b7f77600c71918fa900743cbd510ca2f5fb90875a13e5f8b0466c681b91498"}
Jul 28 01:27:40.258690 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:40.258663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc" event={"ID":"8f263054-4032-44e7-9566-15acc9322453","Type":"ContainerStarted","Data":"5845d93c1e771a554fb7343f3e5930e60518c9a437bfdd6bf5b1eaf9a60e8637"}
Jul 28 01:27:40.266588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:40.266561    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"]
Jul 28 01:27:40.269750 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:40.269725    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc2a0c8_220b_4a5f_8ca9_c8a9382e9d84.slice/crio-b73a6e0eccbda850efd22fb9640062b744b0402a5288ea644b5d854322b7fdda WatchSource:0}: Error finding container b73a6e0eccbda850efd22fb9640062b744b0402a5288ea644b5d854322b7fdda: Status 404 returned error can't find the container with id b73a6e0eccbda850efd22fb9640062b744b0402a5288ea644b5d854322b7fdda
Jul 28 01:27:40.299657 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:40.299608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.26492179 podStartE2EDuration="5.299590846s" totalImagesPullingTime="3.020031958s" totalInitContainerRuntime="14.637098ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:40.299115903 +0000 UTC m=+111.133863451" watchObservedRunningTime="2026-07-28 01:27:40.299590846 +0000 UTC m=+111.134338395"
Jul 28 01:27:41.267050 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:41.267013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k" event={"ID":"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84","Type":"ContainerStarted","Data":"b73a6e0eccbda850efd22fb9640062b744b0402a5288ea644b5d854322b7fdda"}
Jul 28 01:27:42.272379 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:42.272195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc" event={"ID":"8f263054-4032-44e7-9566-15acc9322453","Type":"ContainerStarted","Data":"dd493dbd75226c67139e73894426ee75d1da38638d4f24e20c168503f47bc2f5"}
Jul 28 01:27:42.273914 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:42.273885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k" event={"ID":"9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84","Type":"ContainerStarted","Data":"732fdf0ceec5057e981482bfb5d20f7d6ceb0fc0a7961918c049cc6201e040b7"}
Jul 28 01:27:42.274332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:42.274302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:42.279788 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:42.279772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k"
Jul 28 01:27:42.291600 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:42.291564    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc" podStartSLOduration=2.0093960810000002 podStartE2EDuration="4.291551435s" totalImagesPullingTime="2.282155354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:42.2911469 +0000 UTC m=+113.125894455" watchObservedRunningTime="2026-07-28 01:27:42.291551435 +0000 UTC m=+113.126298981"
Jul 28 01:27:42.310367 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:42.310322    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-5bx4k" podStartSLOduration=1.429003278 podStartE2EDuration="3.310310185s" totalImagesPullingTime="1.881306907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:42.309022191 +0000 UTC m=+113.143769739" watchObservedRunningTime="2026-07-28 01:27:42.310310185 +0000 UTC m=+113.145057769"
Jul 28 01:27:49.301110 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:49.301075    2577 generic.go:356] "Generic (PLEG): container finished" podID="69c6360a-abac-4470-9a52-1eb23a783688" containerID="b5ba7d9522c860f31b56ee54dec20b1b0167a39cf6cef7d74ade2c50f6716418" exitCode=0
Jul 28 01:27:49.301454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:49.301147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s" event={"ID":"69c6360a-abac-4470-9a52-1eb23a783688","Type":"ContainerDied","Data":"b5ba7d9522c860f31b56ee54dec20b1b0167a39cf6cef7d74ade2c50f6716418"}
Jul 28 01:27:49.301454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:49.301414    2577 scope.go:117] "RemoveContainer" containerID="b5ba7d9522c860f31b56ee54dec20b1b0167a39cf6cef7d74ade2c50f6716418"
Jul 28 01:27:50.306520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:50.306481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lrl7s" event={"ID":"69c6360a-abac-4470-9a52-1eb23a783688","Type":"ContainerStarted","Data":"bd74cea55aa9ecdea140ae1e0382b1070bdc414e1e87de302bd4e078b66a830f"}
Jul 28 01:27:55.331555 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:55.331519    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3" containerID="1531615f25edba0338067d2a314dc7c05c4ed529d0c34bc349a9029530a36a10" exitCode=0
Jul 28 01:27:55.331934 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:55.331590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6k54l" event={"ID":"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3","Type":"ContainerDied","Data":"1531615f25edba0338067d2a314dc7c05c4ed529d0c34bc349a9029530a36a10"}
Jul 28 01:27:55.332004 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:55.331988    2577 scope.go:117] "RemoveContainer" containerID="1531615f25edba0338067d2a314dc7c05c4ed529d0c34bc349a9029530a36a10"
Jul 28 01:27:56.337551 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:56.337519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6k54l" event={"ID":"6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3","Type":"ContainerStarted","Data":"fbfdcc6a2d35fad0f43303c1c42f01109d8f811da8af269113d915e093043e10"}
Jul 28 01:27:59.245913 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.245877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:59.246286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.246156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:27:59.405256 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.405231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:27:59.407589 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.407565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8c2454-c3c5-4499-a2d8-1a5cfe353eba-metrics-certs\") pod \"network-metrics-daemon-qbb85\" (UID: \"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba\") " pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:27:59.703671 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.703601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5299b\""
Jul 28 01:27:59.711984 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.711964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbb85"
Jul 28 01:27:59.834559 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:27:59.834531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qbb85"]
Jul 28 01:27:59.837975 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:27:59.837949    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8c2454_c3c5_4499_a2d8_1a5cfe353eba.slice/crio-5d1edf45c23ce2dcb35fd6cb9674c81fe282b3ef68e40ca2b7bed4854caa7913 WatchSource:0}: Error finding container 5d1edf45c23ce2dcb35fd6cb9674c81fe282b3ef68e40ca2b7bed4854caa7913: Status 404 returned error can't find the container with id 5d1edf45c23ce2dcb35fd6cb9674c81fe282b3ef68e40ca2b7bed4854caa7913
Jul 28 01:28:00.355473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:00.355439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbb85" event={"ID":"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba","Type":"ContainerStarted","Data":"5d1edf45c23ce2dcb35fd6cb9674c81fe282b3ef68e40ca2b7bed4854caa7913"}
Jul 28 01:28:01.361311 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:01.361281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbb85" event={"ID":"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba","Type":"ContainerStarted","Data":"67850cabeec748b5763eda6f1e6a27ddac8ebf4517fbe3279ba7dfe5c7c3572e"}
Jul 28 01:28:01.361807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:01.361343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbb85" event={"ID":"fb8c2454-c3c5-4499-a2d8-1a5cfe353eba","Type":"ContainerStarted","Data":"80d85b091de2407f149889858b7b9b2413a30b6dab509908ba4ecb06ec1cd64b"}
Jul 28 01:28:01.396685 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:01.396641    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qbb85" podStartSLOduration=130.492813178 podStartE2EDuration="2m11.396624991s" totalImagesPullingTime="903.811813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:28:01.396302765 +0000 UTC m=+132.231050311" watchObservedRunningTime="2026-07-28 01:28:01.396624991 +0000 UTC m=+132.231372567"
Jul 28 01:28:19.250696 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:19.250665    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:28:19.254648 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:19.254613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c5b67497c-gr5jc"
Jul 28 01:28:58.755722 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.755623    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58476696fd-pthfz"]
Jul 28 01:28:58.759203 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.759179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.761760 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.761734    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 01:28:58.761887 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.761766    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 01:28:58.761887 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.761831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 01:28:58.762069 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.762054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 01:28:58.762069 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.762064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9ggv8\""
Jul 28 01:28:58.762145 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.762103    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 01:28:58.767412 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.767394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 01:28:58.771944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.771924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58476696fd-pthfz"]
Jul 28 01:28:58.890646 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-serving-certs-ca-bundle\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.890826 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-telemeter-client-tls\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.890826 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-secret-telemeter-client\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.890826 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-metrics-client-ca\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.890826 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.890826 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.890826 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sht4n\" (UniqueName: \"kubernetes.io/projected/eb38993c-646f-47cc-a3ac-9759b09df6f6-kube-api-access-sht4n\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.891103 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.890903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-federate-client-tls\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.991833 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-serving-certs-ca-bundle\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-telemeter-client-tls\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-secret-telemeter-client\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-metrics-client-ca\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.991973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sht4n\" (UniqueName: \"kubernetes.io/projected/eb38993c-646f-47cc-a3ac-9759b09df6f6-kube-api-access-sht4n\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992342 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.992252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-federate-client-tls\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992596 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.992574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-serving-certs-ca-bundle\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992839 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.992812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.992950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.992903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb38993c-646f-47cc-a3ac-9759b09df6f6-metrics-client-ca\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.994492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.994470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-telemeter-client-tls\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.994585 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.994562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-federate-client-tls\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.994796 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.994777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-secret-telemeter-client\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.994917 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.994896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb38993c-646f-47cc-a3ac-9759b09df6f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:58.999544 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:58.999526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sht4n\" (UniqueName: \"kubernetes.io/projected/eb38993c-646f-47cc-a3ac-9759b09df6f6-kube-api-access-sht4n\") pod \"telemeter-client-58476696fd-pthfz\" (UID: \"eb38993c-646f-47cc-a3ac-9759b09df6f6\") " pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:59.069255 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:59.069205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz"
Jul 28 01:28:59.211934 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:59.211908    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58476696fd-pthfz"]
Jul 28 01:28:59.213071 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:28:59.213047    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb38993c_646f_47cc_a3ac_9759b09df6f6.slice/crio-f604056fad43471aa1390b684165614090ad5411dc35ad2179f8d20e27a8bd37 WatchSource:0}: Error finding container f604056fad43471aa1390b684165614090ad5411dc35ad2179f8d20e27a8bd37: Status 404 returned error can't find the container with id f604056fad43471aa1390b684165614090ad5411dc35ad2179f8d20e27a8bd37
Jul 28 01:28:59.578024 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:28:59.577993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz" event={"ID":"eb38993c-646f-47cc-a3ac-9759b09df6f6","Type":"ContainerStarted","Data":"f604056fad43471aa1390b684165614090ad5411dc35ad2179f8d20e27a8bd37"}
Jul 28 01:29:01.587614 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:29:01.587580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz" event={"ID":"eb38993c-646f-47cc-a3ac-9759b09df6f6","Type":"ContainerStarted","Data":"ae852d4ecf86a0ba86f934dc780d9267988100b5afd9a64ca0203f74893ab2bb"}
Jul 28 01:29:01.588032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:29:01.587640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz" event={"ID":"eb38993c-646f-47cc-a3ac-9759b09df6f6","Type":"ContainerStarted","Data":"0ec6b35e7ea382fbb442c43687eb412fd3ba1cd6f7c59d58902971fdafc0cf35"}
Jul 28 01:29:01.588032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:29:01.587659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz" event={"ID":"eb38993c-646f-47cc-a3ac-9759b09df6f6","Type":"ContainerStarted","Data":"8217d36b3f4f06ea18a90bf05d1a01f9e41803f08be957c4c060c354f04be3fa"}
Jul 28 01:29:01.616252 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:29:01.616198    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58476696fd-pthfz" podStartSLOduration=2.114341568 podStartE2EDuration="3.616181227s" totalImagesPullingTime="1.501839659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:29:01.613558013 +0000 UTC m=+192.448305574" watchObservedRunningTime="2026-07-28 01:29:01.616181227 +0000 UTC m=+192.450928772"
Jul 28 01:30:35.540576 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.540495    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"]
Jul 28 01:30:35.543900 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.543882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.546367 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.546348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 01:30:35.547299 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.547267    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 01:30:35.547299 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.547277    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jmm2g\""
Jul 28 01:30:35.555568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.555538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"]
Jul 28 01:30:35.652977 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.652942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.652977 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.652979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdm55\" (UniqueName: \"kubernetes.io/projected/1b579399-ae5f-480a-99e3-b77f256c23c2-kube-api-access-rdm55\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.653171 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.653011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.754501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.754476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.754634 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.754506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdm55\" (UniqueName: \"kubernetes.io/projected/1b579399-ae5f-480a-99e3-b77f256c23c2-kube-api-access-rdm55\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.754634 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.754534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.754829 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.754816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.754893 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.754874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.766033 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.766006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdm55\" (UniqueName: \"kubernetes.io/projected/1b579399-ae5f-480a-99e3-b77f256c23c2-kube-api-access-rdm55\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.852371 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.852321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:35.994895 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:35.994848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"]
Jul 28 01:30:35.996930 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:30:35.996902    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b579399_ae5f_480a_99e3_b77f256c23c2.slice/crio-2fc77f9f69e249e534c8a4a3bf49daa0c8f2c1e8073f3594241919288750170c WatchSource:0}: Error finding container 2fc77f9f69e249e534c8a4a3bf49daa0c8f2c1e8073f3594241919288750170c: Status 404 returned error can't find the container with id 2fc77f9f69e249e534c8a4a3bf49daa0c8f2c1e8073f3594241919288750170c
Jul 28 01:30:36.952973 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:36.952935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerStarted","Data":"2fc77f9f69e249e534c8a4a3bf49daa0c8f2c1e8073f3594241919288750170c"}
Jul 28 01:30:41.977068 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:41.977032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerStarted","Data":"9ba941c785d938215735d3ffbbf280f88def6011871acce6a774c793593b6b4a"}
Jul 28 01:30:42.982512 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:42.982480    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerID="9ba941c785d938215735d3ffbbf280f88def6011871acce6a774c793593b6b4a" exitCode=0
Jul 28 01:30:42.982512 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:42.982515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerDied","Data":"9ba941c785d938215735d3ffbbf280f88def6011871acce6a774c793593b6b4a"}
Jul 28 01:30:44.992568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:44.992537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerStarted","Data":"3b57fbcfec271f30f41c277e935c261c493e467b3a7fa6cb81e4bfbb006d19cb"}
Jul 28 01:30:45.998810 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:45.998776    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerID="3b57fbcfec271f30f41c277e935c261c493e467b3a7fa6cb81e4bfbb006d19cb" exitCode=0
Jul 28 01:30:45.999215 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:45.998833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerDied","Data":"3b57fbcfec271f30f41c277e935c261c493e467b3a7fa6cb81e4bfbb006d19cb"}
Jul 28 01:30:51.364521 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:51.364492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:30:51.366911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:51.364535    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:30:51.374727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:51.374710    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:30:52.033016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:52.032980    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerID="fd84b3f19c9d4f29dfc092fdfdafebd9c4248423d6804f7e884356d47671b358" exitCode=0
Jul 28 01:30:52.033189 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:52.033071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerDied","Data":"fd84b3f19c9d4f29dfc092fdfdafebd9c4248423d6804f7e884356d47671b358"}
Jul 28 01:30:53.160971 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.160951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:53.208948 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.208924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-util\") pod \"1b579399-ae5f-480a-99e3-b77f256c23c2\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") "
Jul 28 01:30:53.209058 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.208965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdm55\" (UniqueName: \"kubernetes.io/projected/1b579399-ae5f-480a-99e3-b77f256c23c2-kube-api-access-rdm55\") pod \"1b579399-ae5f-480a-99e3-b77f256c23c2\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") "
Jul 28 01:30:53.209058 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.209000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-bundle\") pod \"1b579399-ae5f-480a-99e3-b77f256c23c2\" (UID: \"1b579399-ae5f-480a-99e3-b77f256c23c2\") "
Jul 28 01:30:53.209633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.209611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-bundle" (OuterVolumeSpecName: "bundle") pod "1b579399-ae5f-480a-99e3-b77f256c23c2" (UID: "1b579399-ae5f-480a-99e3-b77f256c23c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:30:53.211097 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.211064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b579399-ae5f-480a-99e3-b77f256c23c2-kube-api-access-rdm55" (OuterVolumeSpecName: "kube-api-access-rdm55") pod "1b579399-ae5f-480a-99e3-b77f256c23c2" (UID: "1b579399-ae5f-480a-99e3-b77f256c23c2"). InnerVolumeSpecName "kube-api-access-rdm55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:30:53.213715 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.213696    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-util" (OuterVolumeSpecName: "util") pod "1b579399-ae5f-480a-99e3-b77f256c23c2" (UID: "1b579399-ae5f-480a-99e3-b77f256c23c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:30:53.309993 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.309925    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdm55\" (UniqueName: \"kubernetes.io/projected/1b579399-ae5f-480a-99e3-b77f256c23c2-kube-api-access-rdm55\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:30:53.309993 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.309958    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:30:53.309993 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:53.309971    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b579399-ae5f-480a-99e3-b77f256c23c2-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:30:54.048180 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:54.048149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6" event={"ID":"1b579399-ae5f-480a-99e3-b77f256c23c2","Type":"ContainerDied","Data":"2fc77f9f69e249e534c8a4a3bf49daa0c8f2c1e8073f3594241919288750170c"}
Jul 28 01:30:54.048180 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:54.048167    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hbdm6"
Jul 28 01:30:54.048412 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:54.048214    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc77f9f69e249e534c8a4a3bf49daa0c8f2c1e8073f3594241919288750170c"
Jul 28 01:30:58.510966 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.510932    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"]
Jul 28 01:30:58.511321 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511278    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="util"
Jul 28 01:30:58.511321 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511291    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="util"
Jul 28 01:30:58.511321 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511309    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="pull"
Jul 28 01:30:58.511321 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511314    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="pull"
Jul 28 01:30:58.511321 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511323    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="extract"
Jul 28 01:30:58.511492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511331    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="extract"
Jul 28 01:30:58.511492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.511395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b579399-ae5f-480a-99e3-b77f256c23c2" containerName="extract"
Jul 28 01:30:58.539588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.539558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"]
Jul 28 01:30:58.539766 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.539689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.542480 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.542448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 01:30:58.542607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.542451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 01:30:58.542607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.542534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 01:30:58.542607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.542453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-k8gv7\""
Jul 28 01:30:58.654080 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.654048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/fef9b5e4-5f85-4815-8cf4-a65c87272833-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp\" (UID: \"fef9b5e4-5f85-4815-8cf4-a65c87272833\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.654235 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.654096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf74p\" (UniqueName: \"kubernetes.io/projected/fef9b5e4-5f85-4815-8cf4-a65c87272833-kube-api-access-rf74p\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp\" (UID: \"fef9b5e4-5f85-4815-8cf4-a65c87272833\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.754967 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.754936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/fef9b5e4-5f85-4815-8cf4-a65c87272833-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp\" (UID: \"fef9b5e4-5f85-4815-8cf4-a65c87272833\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.755109 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.754977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf74p\" (UniqueName: \"kubernetes.io/projected/fef9b5e4-5f85-4815-8cf4-a65c87272833-kube-api-access-rf74p\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp\" (UID: \"fef9b5e4-5f85-4815-8cf4-a65c87272833\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.757265 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.757241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/fef9b5e4-5f85-4815-8cf4-a65c87272833-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp\" (UID: \"fef9b5e4-5f85-4815-8cf4-a65c87272833\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.763161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.763100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf74p\" (UniqueName: \"kubernetes.io/projected/fef9b5e4-5f85-4815-8cf4-a65c87272833-kube-api-access-rf74p\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp\" (UID: \"fef9b5e4-5f85-4815-8cf4-a65c87272833\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.849796 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.849764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:30:58.998911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:58.998887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"]
Jul 28 01:30:59.001031 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:30:59.001004    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef9b5e4_5f85_4815_8cf4_a65c87272833.slice/crio-1bf540fb79bc1385aad272acba9f28822d93dc715160cb06e4077d7afeb23f06 WatchSource:0}: Error finding container 1bf540fb79bc1385aad272acba9f28822d93dc715160cb06e4077d7afeb23f06: Status 404 returned error can't find the container with id 1bf540fb79bc1385aad272acba9f28822d93dc715160cb06e4077d7afeb23f06
Jul 28 01:30:59.002852 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:59.002835    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:30:59.070108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:30:59.070046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp" event={"ID":"fef9b5e4-5f85-4815-8cf4-a65c87272833","Type":"ContainerStarted","Data":"1bf540fb79bc1385aad272acba9f28822d93dc715160cb06e4077d7afeb23f06"}
Jul 28 01:31:03.089703 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.089660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp" event={"ID":"fef9b5e4-5f85-4815-8cf4-a65c87272833","Type":"ContainerStarted","Data":"a1afaddaac51cd1a24054ae34143285a3713585dbc0ffe33ced4157d2cc8b49c"}
Jul 28 01:31:03.090118 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.089781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:31:03.112496 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.112438    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp" podStartSLOduration=1.6907121059999999 podStartE2EDuration="5.112417164s" totalImagesPullingTime="3.421705058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:03.109934824 +0000 UTC m=+313.944682370" watchObservedRunningTime="2026-07-28 01:31:03.112417164 +0000 UTC m=+313.947164711"
Jul 28 01:31:03.368579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.368495    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"]
Jul 28 01:31:03.389919 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.389887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"]
Jul 28 01:31:03.390065 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.389932    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.392947 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.392624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 01:31:03.392947 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.392823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 01:31:03.392947 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.392894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vtwrz\""
Jul 28 01:31:03.497926 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.497896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9ccd150a-b347-4648-b8d5-62e9faa82c37-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.498111 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.497942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6zz\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-kube-api-access-mn6zz\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.498111 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.497975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-certificates\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.599028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.598998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6zz\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-kube-api-access-mn6zz\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.599221 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.599041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-certificates\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.599221 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.599124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9ccd150a-b347-4648-b8d5-62e9faa82c37-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.599221 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:31:03.599214    2577 secret.go:281] references non-existent secret key: tls.crt
Jul 28 01:31:03.599401 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:31:03.599234    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 01:31:03.599401 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:31:03.599250    2577 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 28 01:31:03.599401 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:31:03.599268    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 28 01:31:03.599401 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:31:03.599317    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-certificates podName:9ccd150a-b347-4648-b8d5-62e9faa82c37 nodeName:}" failed. No retries permitted until 2026-07-28 01:31:04.099302432 +0000 UTC m=+314.934049956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-certificates") pod "keda-metrics-apiserver-55557df8c-w4jlw" (UID: "9ccd150a-b347-4648-b8d5-62e9faa82c37") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 28 01:31:03.599615 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.599565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9ccd150a-b347-4648-b8d5-62e9faa82c37-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:03.609449 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:03.609422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6zz\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-kube-api-access-mn6zz\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:04.103476 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:04.103447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-certificates\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:04.106436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:04.106418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9ccd150a-b347-4648-b8d5-62e9faa82c37-certificates\") pod \"keda-metrics-apiserver-55557df8c-w4jlw\" (UID: \"9ccd150a-b347-4648-b8d5-62e9faa82c37\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:04.302299 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:04.302269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:04.452775 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:04.452749    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"]
Jul 28 01:31:04.455219 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:31:04.455186    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccd150a_b347_4648_b8d5_62e9faa82c37.slice/crio-e07cf09d73f5120233d8f86400ecbb4aaa0309a3a6443e1b8fb3f53290e84694 WatchSource:0}: Error finding container e07cf09d73f5120233d8f86400ecbb4aaa0309a3a6443e1b8fb3f53290e84694: Status 404 returned error can't find the container with id e07cf09d73f5120233d8f86400ecbb4aaa0309a3a6443e1b8fb3f53290e84694
Jul 28 01:31:05.099036 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:05.099000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw" event={"ID":"9ccd150a-b347-4648-b8d5-62e9faa82c37","Type":"ContainerStarted","Data":"e07cf09d73f5120233d8f86400ecbb4aaa0309a3a6443e1b8fb3f53290e84694"}
Jul 28 01:31:08.116875 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:08.116838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw" event={"ID":"9ccd150a-b347-4648-b8d5-62e9faa82c37","Type":"ContainerStarted","Data":"72c3ba1c97d7f45e57c9c3d88ddc8689aafbea3872b2de608d280b101fe3a2fb"}
Jul 28 01:31:08.117253 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:08.117002    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:08.137574 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:08.137523    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw" podStartSLOduration=2.234715449 podStartE2EDuration="5.137509742s" totalImagesPullingTime="2.902794293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:08.135489512 +0000 UTC m=+318.970237061" watchObservedRunningTime="2026-07-28 01:31:08.137509742 +0000 UTC m=+318.972257288"
Jul 28 01:31:19.125619 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:19.125590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-w4jlw"
Jul 28 01:31:24.096125 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:31:24.096098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-fz5pp"
Jul 28 01:32:21.441710 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.441673    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-5lpl7"]
Jul 28 01:32:21.444972 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.444954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:21.447680 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.447662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 01:32:21.447784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.447667    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 01:32:21.448267 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.448242    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 01:32:21.448706 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.448691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7btx6\""
Jul 28 01:32:21.457052 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.457031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-5lpl7"]
Jul 28 01:32:21.466201 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.466176    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7584bddd84-xc2hz"]
Jul 28 01:32:21.469394 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.469379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.471987 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.471968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 01:32:21.472081 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.472070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-pmsft\""
Jul 28 01:32:21.485296 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.485276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7584bddd84-xc2hz"]
Jul 28 01:32:21.545796 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.545766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2zl\" (UniqueName: \"kubernetes.io/projected/54a42542-497f-4e37-af12-43c625455789-kube-api-access-5q2zl\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:21.545928 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.545894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:21.646631 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.646596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:21.646768 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.646649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2zl\" (UniqueName: \"kubernetes.io/projected/54a42542-497f-4e37-af12-43c625455789-kube-api-access-5q2zl\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:21.646768 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.646703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhj2d\" (UniqueName: \"kubernetes.io/projected/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-kube-api-access-hhj2d\") pod \"llmisvc-controller-manager-7584bddd84-xc2hz\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") " pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.646768 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:32:21.646733    2577 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 01:32:21.646953 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.646788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-cert\") pod \"llmisvc-controller-manager-7584bddd84-xc2hz\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") " pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.646953 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:32:21.646809    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert podName:54a42542-497f-4e37-af12-43c625455789 nodeName:}" failed. No retries permitted until 2026-07-28 01:32:22.146786748 +0000 UTC m=+392.981534290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert") pod "kserve-controller-manager-ff96d5959-5lpl7" (UID: "54a42542-497f-4e37-af12-43c625455789") : secret "kserve-webhook-server-cert" not found
Jul 28 01:32:21.656684 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.656655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2zl\" (UniqueName: \"kubernetes.io/projected/54a42542-497f-4e37-af12-43c625455789-kube-api-access-5q2zl\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:21.747456 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.747434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhj2d\" (UniqueName: \"kubernetes.io/projected/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-kube-api-access-hhj2d\") pod \"llmisvc-controller-manager-7584bddd84-xc2hz\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") " pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.747566 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.747476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-cert\") pod \"llmisvc-controller-manager-7584bddd84-xc2hz\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") " pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.749686 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.749669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-cert\") pod \"llmisvc-controller-manager-7584bddd84-xc2hz\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") " pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.761893 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.761851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhj2d\" (UniqueName: \"kubernetes.io/projected/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-kube-api-access-hhj2d\") pod \"llmisvc-controller-manager-7584bddd84-xc2hz\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") " pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.779702 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.779685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:21.910472 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:21.910430    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7584bddd84-xc2hz"]
Jul 28 01:32:21.912686 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:32:21.912651    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7e5a9ad_6752_4e1e_bdcc_96757a77bd85.slice/crio-0e04adfb54aa7eee83d751bb1f0b139d0123159683f25307fd1379e6a3367b87 WatchSource:0}: Error finding container 0e04adfb54aa7eee83d751bb1f0b139d0123159683f25307fd1379e6a3367b87: Status 404 returned error can't find the container with id 0e04adfb54aa7eee83d751bb1f0b139d0123159683f25307fd1379e6a3367b87
Jul 28 01:32:22.150700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:22.150621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:22.153040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:22.153016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert\") pod \"kserve-controller-manager-ff96d5959-5lpl7\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") " pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:22.358450 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:22.358419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:22.428781 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:22.428754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz" event={"ID":"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85","Type":"ContainerStarted","Data":"0e04adfb54aa7eee83d751bb1f0b139d0123159683f25307fd1379e6a3367b87"}
Jul 28 01:32:22.487325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:22.487301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-5lpl7"]
Jul 28 01:32:22.489749 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:32:22.489719    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a42542_497f_4e37_af12_43c625455789.slice/crio-d97f26de41ba9a9dbeac2005277fc7d39ceb61e376d754117ec4752c7de327f0 WatchSource:0}: Error finding container d97f26de41ba9a9dbeac2005277fc7d39ceb61e376d754117ec4752c7de327f0: Status 404 returned error can't find the container with id d97f26de41ba9a9dbeac2005277fc7d39ceb61e376d754117ec4752c7de327f0
Jul 28 01:32:23.438348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:23.438218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7" event={"ID":"54a42542-497f-4e37-af12-43c625455789","Type":"ContainerStarted","Data":"d97f26de41ba9a9dbeac2005277fc7d39ceb61e376d754117ec4752c7de327f0"}
Jul 28 01:32:26.455401 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:26.455361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz" event={"ID":"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85","Type":"ContainerStarted","Data":"edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324"}
Jul 28 01:32:26.455814 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:26.455513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:26.456879 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:26.456837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7" event={"ID":"54a42542-497f-4e37-af12-43c625455789","Type":"ContainerStarted","Data":"87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5"}
Jul 28 01:32:26.456988 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:26.456979    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:32:26.476487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:26.476448    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz" podStartSLOduration=1.583408469 podStartE2EDuration="5.476436453s" totalImagesPullingTime="3.893027984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:26.476254987 +0000 UTC m=+397.311002535" watchObservedRunningTime="2026-07-28 01:32:26.476436453 +0000 UTC m=+397.311183996"
Jul 28 01:32:26.501143 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:26.501104    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7" podStartSLOduration=2.185679394 podStartE2EDuration="5.501092611s" totalImagesPullingTime="3.315413217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:26.499258239 +0000 UTC m=+397.334005785" watchObservedRunningTime="2026-07-28 01:32:26.501092611 +0000 UTC m=+397.335840157"
Jul 28 01:32:57.462844 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:57.462804    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 01:32:57.465840 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:32:57.465821    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:33:05.697097 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.697064    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"]
Jul 28 01:33:05.703609 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.703591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.706386 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.706357    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 01:33:05.706539 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.706440    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 01:33:05.707377 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.707354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jmm2g\""
Jul 28 01:33:05.710131 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.710099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"]
Jul 28 01:33:05.812777 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.812744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.812777 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.812781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.813011 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.812915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfdz\" (UniqueName: \"kubernetes.io/projected/3011b990-d198-40c6-a374-90842bd324dd-kube-api-access-vlfdz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.913818 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.913779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.914009 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.913823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.914009 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.913928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfdz\" (UniqueName: \"kubernetes.io/projected/3011b990-d198-40c6-a374-90842bd324dd-kube-api-access-vlfdz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.914168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.914145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.914231 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.914175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:05.922553 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:05.922524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfdz\" (UniqueName: \"kubernetes.io/projected/3011b990-d198-40c6-a374-90842bd324dd-kube-api-access-vlfdz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:06.014231 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:06.014192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:06.140200 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:06.140176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"]
Jul 28 01:33:06.142295 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:33:06.142269    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3011b990_d198_40c6_a374_90842bd324dd.slice/crio-7b8e38199560dbcb8b7dcde1a5cfb77956434789dd0ed31599b18b9f6b4d8579 WatchSource:0}: Error finding container 7b8e38199560dbcb8b7dcde1a5cfb77956434789dd0ed31599b18b9f6b4d8579: Status 404 returned error can't find the container with id 7b8e38199560dbcb8b7dcde1a5cfb77956434789dd0ed31599b18b9f6b4d8579
Jul 28 01:33:06.636846 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:06.636815    2577 generic.go:356] "Generic (PLEG): container finished" podID="3011b990-d198-40c6-a374-90842bd324dd" containerID="f14a47dce5ce090dbfc988412a660324bcccf69cbb275c803eb3f436632f9104" exitCode=0
Jul 28 01:33:06.637040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:06.636893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk" event={"ID":"3011b990-d198-40c6-a374-90842bd324dd","Type":"ContainerDied","Data":"f14a47dce5ce090dbfc988412a660324bcccf69cbb275c803eb3f436632f9104"}
Jul 28 01:33:06.637040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:06.636940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk" event={"ID":"3011b990-d198-40c6-a374-90842bd324dd","Type":"ContainerStarted","Data":"7b8e38199560dbcb8b7dcde1a5cfb77956434789dd0ed31599b18b9f6b4d8579"}
Jul 28 01:33:09.654221 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:09.654189    2577 generic.go:356] "Generic (PLEG): container finished" podID="3011b990-d198-40c6-a374-90842bd324dd" containerID="ed246e11597f72df0f9d39c995ccf1979e55e3255a7687a4c3dcdbaa093fd02d" exitCode=0
Jul 28 01:33:09.654619 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:09.654247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk" event={"ID":"3011b990-d198-40c6-a374-90842bd324dd","Type":"ContainerDied","Data":"ed246e11597f72df0f9d39c995ccf1979e55e3255a7687a4c3dcdbaa093fd02d"}
Jul 28 01:33:10.661017 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:10.660981    2577 generic.go:356] "Generic (PLEG): container finished" podID="3011b990-d198-40c6-a374-90842bd324dd" containerID="2278277e375b8166aea992472a28f8edd0ff38ca73d1cfd706e87630cc296371" exitCode=0
Jul 28 01:33:10.661362 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:10.661041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk" event={"ID":"3011b990-d198-40c6-a374-90842bd324dd","Type":"ContainerDied","Data":"2278277e375b8166aea992472a28f8edd0ff38ca73d1cfd706e87630cc296371"}
Jul 28 01:33:11.792538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.792516    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:11.965821 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.965751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlfdz\" (UniqueName: \"kubernetes.io/projected/3011b990-d198-40c6-a374-90842bd324dd-kube-api-access-vlfdz\") pod \"3011b990-d198-40c6-a374-90842bd324dd\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") "
Jul 28 01:33:11.965821 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.965811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-util\") pod \"3011b990-d198-40c6-a374-90842bd324dd\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") "
Jul 28 01:33:11.966019 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.965885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-bundle\") pod \"3011b990-d198-40c6-a374-90842bd324dd\" (UID: \"3011b990-d198-40c6-a374-90842bd324dd\") "
Jul 28 01:33:11.966521 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.966494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-bundle" (OuterVolumeSpecName: "bundle") pod "3011b990-d198-40c6-a374-90842bd324dd" (UID: "3011b990-d198-40c6-a374-90842bd324dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:11.967763 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.967740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3011b990-d198-40c6-a374-90842bd324dd-kube-api-access-vlfdz" (OuterVolumeSpecName: "kube-api-access-vlfdz") pod "3011b990-d198-40c6-a374-90842bd324dd" (UID: "3011b990-d198-40c6-a374-90842bd324dd"). InnerVolumeSpecName "kube-api-access-vlfdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:33:11.970580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:11.970561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-util" (OuterVolumeSpecName: "util") pod "3011b990-d198-40c6-a374-90842bd324dd" (UID: "3011b990-d198-40c6-a374-90842bd324dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:12.066627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:12.066603    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:33:12.066627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:12.066623    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3011b990-d198-40c6-a374-90842bd324dd-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:33:12.066768 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:12.066632    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfdz\" (UniqueName: \"kubernetes.io/projected/3011b990-d198-40c6-a374-90842bd324dd-kube-api-access-vlfdz\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:33:12.671161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:12.671132    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk"
Jul 28 01:33:12.671317 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:12.671134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540wnsdk" event={"ID":"3011b990-d198-40c6-a374-90842bd324dd","Type":"ContainerDied","Data":"7b8e38199560dbcb8b7dcde1a5cfb77956434789dd0ed31599b18b9f6b4d8579"}
Jul 28 01:33:12.671317 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:12.671253    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8e38199560dbcb8b7dcde1a5cfb77956434789dd0ed31599b18b9f6b4d8579"
Jul 28 01:33:27.628877 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.628835    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"]
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629158    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="extract"
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629169    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="extract"
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629182    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="util"
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629187    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="util"
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="pull"
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629202    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="pull"
Jul 28 01:33:27.629254 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.629257    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3011b990-d198-40c6-a374-90842bd324dd" containerName="extract"
Jul 28 01:33:27.634732 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.634716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.637262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.637233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 01:33:27.638070 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.638052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jmm2g\""
Jul 28 01:33:27.638173 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.638071    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 01:33:27.644153 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.644126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"]
Jul 28 01:33:27.690051 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.690021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.690168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.690053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7t4\" (UniqueName: \"kubernetes.io/projected/d2d28bd3-336c-4fc2-b099-8b87f582d47b-kube-api-access-tx7t4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.690168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.690145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.790950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.790921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.791102 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.791046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.791102 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.791097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7t4\" (UniqueName: \"kubernetes.io/projected/d2d28bd3-336c-4fc2-b099-8b87f582d47b-kube-api-access-tx7t4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.791370 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.791343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.791516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.791497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.799865 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.799842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7t4\" (UniqueName: \"kubernetes.io/projected/d2d28bd3-336c-4fc2-b099-8b87f582d47b-kube-api-access-tx7t4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:27.950843 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:27.950752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:28.081104 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:28.081079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"]
Jul 28 01:33:28.083277 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:33:28.083250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d28bd3_336c_4fc2_b099_8b87f582d47b.slice/crio-d270dfb0b43f0c1056170f101de4731189d3cd91b94610534cc664a40f92d182 WatchSource:0}: Error finding container d270dfb0b43f0c1056170f101de4731189d3cd91b94610534cc664a40f92d182: Status 404 returned error can't find the container with id d270dfb0b43f0c1056170f101de4731189d3cd91b94610534cc664a40f92d182
Jul 28 01:33:28.741473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:28.741439    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerID="442bdaa65a7b1038842555e00a07a5f5cb8ce9399503d5cbd7922363b957b429" exitCode=0
Jul 28 01:33:28.741473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:28.741476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg" event={"ID":"d2d28bd3-336c-4fc2-b099-8b87f582d47b","Type":"ContainerDied","Data":"442bdaa65a7b1038842555e00a07a5f5cb8ce9399503d5cbd7922363b957b429"}
Jul 28 01:33:28.742019 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:28.741509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg" event={"ID":"d2d28bd3-336c-4fc2-b099-8b87f582d47b","Type":"ContainerStarted","Data":"d270dfb0b43f0c1056170f101de4731189d3cd91b94610534cc664a40f92d182"}
Jul 28 01:33:31.756964 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:31.756928    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerID="6f65451c14440df7bc8973566e859dc1cff058e6cc5762cada4eca0bbe224a80" exitCode=0
Jul 28 01:33:31.757328 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:31.756976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg" event={"ID":"d2d28bd3-336c-4fc2-b099-8b87f582d47b","Type":"ContainerDied","Data":"6f65451c14440df7bc8973566e859dc1cff058e6cc5762cada4eca0bbe224a80"}
Jul 28 01:33:32.762956 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:32.762923    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerID="d3940b2f03ff02a99040053c0ebea24a7f559b5ec6cb64fcd8cb695403af41b1" exitCode=0
Jul 28 01:33:32.762956 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:32.762958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg" event={"ID":"d2d28bd3-336c-4fc2-b099-8b87f582d47b","Type":"ContainerDied","Data":"d3940b2f03ff02a99040053c0ebea24a7f559b5ec6cb64fcd8cb695403af41b1"}
Jul 28 01:33:33.898682 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.898657    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:33.941738 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.941715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx7t4\" (UniqueName: \"kubernetes.io/projected/d2d28bd3-336c-4fc2-b099-8b87f582d47b-kube-api-access-tx7t4\") pod \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") "
Jul 28 01:33:33.941902 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.941796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-bundle\") pod \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") "
Jul 28 01:33:33.941902 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.941843    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-util\") pod \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\" (UID: \"d2d28bd3-336c-4fc2-b099-8b87f582d47b\") "
Jul 28 01:33:33.942190 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.942159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-bundle" (OuterVolumeSpecName: "bundle") pod "d2d28bd3-336c-4fc2-b099-8b87f582d47b" (UID: "d2d28bd3-336c-4fc2-b099-8b87f582d47b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:33.943721 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.943697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d28bd3-336c-4fc2-b099-8b87f582d47b-kube-api-access-tx7t4" (OuterVolumeSpecName: "kube-api-access-tx7t4") pod "d2d28bd3-336c-4fc2-b099-8b87f582d47b" (UID: "d2d28bd3-336c-4fc2-b099-8b87f582d47b"). InnerVolumeSpecName "kube-api-access-tx7t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:33:33.949022 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:33.948994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-util" (OuterVolumeSpecName: "util") pod "d2d28bd3-336c-4fc2-b099-8b87f582d47b" (UID: "d2d28bd3-336c-4fc2-b099-8b87f582d47b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:34.042893 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:34.042807    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:33:34.042893 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:34.042834    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7t4\" (UniqueName: \"kubernetes.io/projected/d2d28bd3-336c-4fc2-b099-8b87f582d47b-kube-api-access-tx7t4\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:33:34.042893 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:34.042843    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2d28bd3-336c-4fc2-b099-8b87f582d47b-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:33:34.773565 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:34.773535    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg"
Jul 28 01:33:34.773565 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:34.773545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb8kwg" event={"ID":"d2d28bd3-336c-4fc2-b099-8b87f582d47b","Type":"ContainerDied","Data":"d270dfb0b43f0c1056170f101de4731189d3cd91b94610534cc664a40f92d182"}
Jul 28 01:33:34.773758 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:34.773585    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d270dfb0b43f0c1056170f101de4731189d3cd91b94610534cc664a40f92d182"
Jul 28 01:33:55.762157 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762123    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"]
Jul 28 01:33:55.762501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="extract"
Jul 28 01:33:55.762501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762450    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="extract"
Jul 28 01:33:55.762501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762465    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="pull"
Jul 28 01:33:55.762501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762470    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="pull"
Jul 28 01:33:55.762501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762479    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="util"
Jul 28 01:33:55.762501 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762485    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="util"
Jul 28 01:33:55.762677 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.762536    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d28bd3-336c-4fc2-b099-8b87f582d47b" containerName="extract"
Jul 28 01:33:55.768832 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.768814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.771274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.771254    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 01:33:55.771274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.771263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 01:33:55.771640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.771616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 01:33:55.771751 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.771624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gxwfh\""
Jul 28 01:33:55.771927 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.771910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 01:33:55.772013 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.771944    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 01:33:55.772326 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.772311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 01:33:55.782273 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.782254    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"]
Jul 28 01:33:55.936049 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.936049 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.936281 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.936281 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.936281 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3aed44-1564-4f05-a8a8-2f3b536464a2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.936432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:55.936432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:55.936319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgn4n\" (UniqueName: \"kubernetes.io/projected/bc3aed44-1564-4f05-a8a8-2f3b536464a2-kube-api-access-dgn4n\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037058 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.036976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037058 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037058 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3aed44-1564-4f05-a8a8-2f3b536464a2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn4n\" (UniqueName: \"kubernetes.io/projected/bc3aed44-1564-4f05-a8a8-2f3b536464a2-kube-api-access-dgn4n\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.037924 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.037897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.039343 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.039310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc3aed44-1564-4f05-a8a8-2f3b536464a2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.039532 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.039517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.039598 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.039512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.039598 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.039574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.045167 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.045146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn4n\" (UniqueName: \"kubernetes.io/projected/bc3aed44-1564-4f05-a8a8-2f3b536464a2-kube-api-access-dgn4n\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.045256 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.045199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc3aed44-1564-4f05-a8a8-2f3b536464a2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4r9gw\" (UID: \"bc3aed44-1564-4f05-a8a8-2f3b536464a2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.078744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.078723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:56.220626 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.220599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"]
Jul 28 01:33:56.222566 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:33:56.222540    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3aed44_1564_4f05_a8a8_2f3b536464a2.slice/crio-ce959e090e35b745ac46a2759b811a044cb2f4e2c65d80ea33e2eae0f852d406 WatchSource:0}: Error finding container ce959e090e35b745ac46a2759b811a044cb2f4e2c65d80ea33e2eae0f852d406: Status 404 returned error can't find the container with id ce959e090e35b745ac46a2759b811a044cb2f4e2c65d80ea33e2eae0f852d406
Jul 28 01:33:56.872076 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:56.872042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw" event={"ID":"bc3aed44-1564-4f05-a8a8-2f3b536464a2","Type":"ContainerStarted","Data":"ce959e090e35b745ac46a2759b811a044cb2f4e2c65d80ea33e2eae0f852d406"}
Jul 28 01:33:58.677335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:58.677288    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 01:33:58.677588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:58.677370    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Jul 28 01:33:58.885727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:58.885693    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw" event={"ID":"bc3aed44-1564-4f05-a8a8-2f3b536464a2","Type":"ContainerStarted","Data":"4d0516042ac7df1e7ec807a1f02691e7d7acedc64d7cc2b36253b4b100c48309"}
Jul 28 01:33:58.885896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:58.885819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:33:58.909043 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:58.908999    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw" podStartSLOduration=1.456514591 podStartE2EDuration="3.908985454s" totalImagesPullingTime="2.452470863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:58.907437563 +0000 UTC m=+489.742185213" watchObservedRunningTime="2026-07-28 01:33:58.908985454 +0000 UTC m=+489.743732999"
Jul 28 01:33:59.891578 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:59.891543    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-4r9gw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 01:33:59.892033 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:33:59.891611    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw" podUID="bc3aed44-1564-4f05-a8a8-2f3b536464a2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:34:02.891280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:02.891251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4r9gw"
Jul 28 01:34:15.661491 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.661457    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"]
Jul 28 01:34:15.664805 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.664790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.667312 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.667288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 01:34:15.667439 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.667319    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 01:34:15.667439 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.667337    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jmm2g\""
Jul 28 01:34:15.673023 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.673000    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"]
Jul 28 01:34:15.700507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.700479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wms5\" (UniqueName: \"kubernetes.io/projected/e9713c38-ad44-419e-a12b-18839cc5401f-kube-api-access-8wms5\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.700648 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.700544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.700648 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.700592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.760316 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.760286    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"]
Jul 28 01:34:15.763749 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.763733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.772168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.772147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"]
Jul 28 01:34:15.801380 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.801492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wms5\" (UniqueName: \"kubernetes.io/projected/e9713c38-ad44-419e-a12b-18839cc5401f-kube-api-access-8wms5\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.801492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.801492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wxf\" (UniqueName: \"kubernetes.io/projected/a68f5fb9-5801-4a5b-a256-60fd12c33817-kube-api-access-f8wxf\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.801627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.801627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.801939 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.802010 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.801953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.811011 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.810989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wms5\" (UniqueName: \"kubernetes.io/projected/e9713c38-ad44-419e-a12b-18839cc5401f-kube-api-access-8wms5\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:15.858623 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.858592    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"]
Jul 28 01:34:15.862160 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.862142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:15.870574 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.870539    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"]
Jul 28 01:34:15.902124 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.902300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wxf\" (UniqueName: \"kubernetes.io/projected/a68f5fb9-5801-4a5b-a256-60fd12c33817-kube-api-access-f8wxf\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.902300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:15.902300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:15.902300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m4dr\" (UniqueName: \"kubernetes.io/projected/b58f9679-a285-48e2-a3b2-2decdad76b1b-kube-api-access-8m4dr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:15.902453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.902453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.902634 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.902617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.910178 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.910156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wxf\" (UniqueName: \"kubernetes.io/projected/a68f5fb9-5801-4a5b-a256-60fd12c33817-kube-api-access-f8wxf\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:15.959270 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.959198    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"]
Jul 28 01:34:15.962793 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.962778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:15.971076 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.971052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"]
Jul 28 01:34:15.974027 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:15.974005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:16.002880 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.002840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.002999 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.002896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.003045 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.002996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.003084 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.003049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbc25\" (UniqueName: \"kubernetes.io/projected/eaf992f3-59df-421b-935f-7a08c41ce467-kube-api-access-kbc25\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.003119 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.003104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.003239 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.003216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m4dr\" (UniqueName: \"kubernetes.io/projected/b58f9679-a285-48e2-a3b2-2decdad76b1b-kube-api-access-8m4dr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.003368 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.003352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.003424 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.003409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.012173 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.012147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m4dr\" (UniqueName: \"kubernetes.io/projected/b58f9679-a285-48e2-a3b2-2decdad76b1b-kube-api-access-8m4dr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.074235 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.074204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:16.103856 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.103817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.104023 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.103952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbc25\" (UniqueName: \"kubernetes.io/projected/eaf992f3-59df-421b-935f-7a08c41ce467-kube-api-access-kbc25\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.104091 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.104063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.104260 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.104238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.104387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.104365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.111968 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.111834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"]
Jul 28 01:34:16.114187 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.114155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbc25\" (UniqueName: \"kubernetes.io/projected/eaf992f3-59df-421b-935f-7a08c41ce467-kube-api-access-kbc25\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.114657 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:16.114635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9713c38_ad44_419e_a12b_18839cc5401f.slice/crio-cd09048894431660373f91b56305451bd475e5aea31ad67821598adee16ab2f7 WatchSource:0}: Error finding container cd09048894431660373f91b56305451bd475e5aea31ad67821598adee16ab2f7: Status 404 returned error can't find the container with id cd09048894431660373f91b56305451bd475e5aea31ad67821598adee16ab2f7
Jul 28 01:34:16.172689 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.172645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:16.238311 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.238280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"]
Jul 28 01:34:16.242926 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:16.242837    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68f5fb9_5801_4a5b_a256_60fd12c33817.slice/crio-1c4b4fd2d94756363049e5a16e8e42da60d9911e456779eddac0fe70ff6dfc4b WatchSource:0}: Error finding container 1c4b4fd2d94756363049e5a16e8e42da60d9911e456779eddac0fe70ff6dfc4b: Status 404 returned error can't find the container with id 1c4b4fd2d94756363049e5a16e8e42da60d9911e456779eddac0fe70ff6dfc4b
Jul 28 01:34:16.274216 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.274195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:16.318751 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.318727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"]
Jul 28 01:34:16.320801 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:16.320772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58f9679_a285_48e2_a3b2_2decdad76b1b.slice/crio-458f2204d5da2883a4bb0e3d928671451f736a14ff1f9badd85b00914aa2e4bd WatchSource:0}: Error finding container 458f2204d5da2883a4bb0e3d928671451f736a14ff1f9badd85b00914aa2e4bd: Status 404 returned error can't find the container with id 458f2204d5da2883a4bb0e3d928671451f736a14ff1f9badd85b00914aa2e4bd
Jul 28 01:34:16.418124 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.418089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"]
Jul 28 01:34:16.420585 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:16.420558    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf992f3_59df_421b_935f_7a08c41ce467.slice/crio-41f534b3f3bdd9bad4f90b37c93b51dc7225c47f1ccef2f6917cb1d642771b32 WatchSource:0}: Error finding container 41f534b3f3bdd9bad4f90b37c93b51dc7225c47f1ccef2f6917cb1d642771b32: Status 404 returned error can't find the container with id 41f534b3f3bdd9bad4f90b37c93b51dc7225c47f1ccef2f6917cb1d642771b32
Jul 28 01:34:16.980556 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.980524    2577 generic.go:356] "Generic (PLEG): container finished" podID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerID="0a3a84bb4caeb15bd2a367d790981e5c884eae738271ed11796317e154f3c397" exitCode=0
Jul 28 01:34:16.981010 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.980605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss" event={"ID":"b58f9679-a285-48e2-a3b2-2decdad76b1b","Type":"ContainerDied","Data":"0a3a84bb4caeb15bd2a367d790981e5c884eae738271ed11796317e154f3c397"}
Jul 28 01:34:16.981010 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.980703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss" event={"ID":"b58f9679-a285-48e2-a3b2-2decdad76b1b","Type":"ContainerStarted","Data":"458f2204d5da2883a4bb0e3d928671451f736a14ff1f9badd85b00914aa2e4bd"}
Jul 28 01:34:16.982205 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.982182    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9713c38-ad44-419e-a12b-18839cc5401f" containerID="f7a349ac7a8f5f7f6d0f06ba9c03dfa16e5d70ade8eee20b6c7264dd46ff8ad0" exitCode=0
Jul 28 01:34:16.982286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.982257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt" event={"ID":"e9713c38-ad44-419e-a12b-18839cc5401f","Type":"ContainerDied","Data":"f7a349ac7a8f5f7f6d0f06ba9c03dfa16e5d70ade8eee20b6c7264dd46ff8ad0"}
Jul 28 01:34:16.982356 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.982316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt" event={"ID":"e9713c38-ad44-419e-a12b-18839cc5401f","Type":"ContainerStarted","Data":"cd09048894431660373f91b56305451bd475e5aea31ad67821598adee16ab2f7"}
Jul 28 01:34:16.983749 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.983725    2577 generic.go:356] "Generic (PLEG): container finished" podID="eaf992f3-59df-421b-935f-7a08c41ce467" containerID="6ceb06be5fa045af78fe2b327e7207a25902c3405a49257cffad71cc190e6ded" exitCode=0
Jul 28 01:34:16.983878 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.983835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m" event={"ID":"eaf992f3-59df-421b-935f-7a08c41ce467","Type":"ContainerDied","Data":"6ceb06be5fa045af78fe2b327e7207a25902c3405a49257cffad71cc190e6ded"}
Jul 28 01:34:16.983953 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.983921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m" event={"ID":"eaf992f3-59df-421b-935f-7a08c41ce467","Type":"ContainerStarted","Data":"41f534b3f3bdd9bad4f90b37c93b51dc7225c47f1ccef2f6917cb1d642771b32"}
Jul 28 01:34:16.985482 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.985458    2577 generic.go:356] "Generic (PLEG): container finished" podID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerID="15b8e19e401f12265912b35638ca5b5dd801a0088f62f5643d8fc6d01d88ddf7" exitCode=0
Jul 28 01:34:16.985577 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.985542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28" event={"ID":"a68f5fb9-5801-4a5b-a256-60fd12c33817","Type":"ContainerDied","Data":"15b8e19e401f12265912b35638ca5b5dd801a0088f62f5643d8fc6d01d88ddf7"}
Jul 28 01:34:16.985622 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:16.985604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28" event={"ID":"a68f5fb9-5801-4a5b-a256-60fd12c33817","Type":"ContainerStarted","Data":"1c4b4fd2d94756363049e5a16e8e42da60d9911e456779eddac0fe70ff6dfc4b"}
Jul 28 01:34:17.992781 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:17.992743    2577 generic.go:356] "Generic (PLEG): container finished" podID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerID="a7a50c4155395674ee646b9275684c47222683a8c4e8fa9e496a2b57882c9774" exitCode=0
Jul 28 01:34:17.993216 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:17.992783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss" event={"ID":"b58f9679-a285-48e2-a3b2-2decdad76b1b","Type":"ContainerDied","Data":"a7a50c4155395674ee646b9275684c47222683a8c4e8fa9e496a2b57882c9774"}
Jul 28 01:34:18.999338 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:18.999307    2577 generic.go:356] "Generic (PLEG): container finished" podID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerID="f04e71127bbe4998afac172e44bee08565bdc8d0526b1b8db554d2192c84e16e" exitCode=0
Jul 28 01:34:18.999684 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:18.999381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss" event={"ID":"b58f9679-a285-48e2-a3b2-2decdad76b1b","Type":"ContainerDied","Data":"f04e71127bbe4998afac172e44bee08565bdc8d0526b1b8db554d2192c84e16e"}
Jul 28 01:34:19.001077 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:19.001056    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9713c38-ad44-419e-a12b-18839cc5401f" containerID="5f9d5807b4f6a9795131b778adde64cb18f59f6f6b8a810906f2f26dd5983d48" exitCode=0
Jul 28 01:34:19.001173 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:19.001125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt" event={"ID":"e9713c38-ad44-419e-a12b-18839cc5401f","Type":"ContainerDied","Data":"5f9d5807b4f6a9795131b778adde64cb18f59f6f6b8a810906f2f26dd5983d48"}
Jul 28 01:34:19.002997 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:19.002971    2577 generic.go:356] "Generic (PLEG): container finished" podID="eaf992f3-59df-421b-935f-7a08c41ce467" containerID="9e0d21314e93cd2c48837e0bab0591aced9232de95aa7f2f6d26e9d179fb5587" exitCode=0
Jul 28 01:34:19.003093 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:19.003054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m" event={"ID":"eaf992f3-59df-421b-935f-7a08c41ce467","Type":"ContainerDied","Data":"9e0d21314e93cd2c48837e0bab0591aced9232de95aa7f2f6d26e9d179fb5587"}
Jul 28 01:34:19.004999 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:19.004980    2577 generic.go:356] "Generic (PLEG): container finished" podID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerID="0e199cb3c7a0bce10bdef7392074684009d1a6d297eee39b2e829e863f9f519a" exitCode=0
Jul 28 01:34:19.005072 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:19.005007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28" event={"ID":"a68f5fb9-5801-4a5b-a256-60fd12c33817","Type":"ContainerDied","Data":"0e199cb3c7a0bce10bdef7392074684009d1a6d297eee39b2e829e863f9f519a"}
Jul 28 01:34:20.012292 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.012257    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9713c38-ad44-419e-a12b-18839cc5401f" containerID="006e204967d435e169b2ca829499a6decf4667c83af35835aabae08742b55c85" exitCode=0
Jul 28 01:34:20.012708 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.012337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt" event={"ID":"e9713c38-ad44-419e-a12b-18839cc5401f","Type":"ContainerDied","Data":"006e204967d435e169b2ca829499a6decf4667c83af35835aabae08742b55c85"}
Jul 28 01:34:20.014391 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.014368    2577 generic.go:356] "Generic (PLEG): container finished" podID="eaf992f3-59df-421b-935f-7a08c41ce467" containerID="ce4ef4cf8c12739bb272c9a25c946f4d10de012756cc7a3138dafcce0520b87b" exitCode=0
Jul 28 01:34:20.014527 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.014454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m" event={"ID":"eaf992f3-59df-421b-935f-7a08c41ce467","Type":"ContainerDied","Data":"ce4ef4cf8c12739bb272c9a25c946f4d10de012756cc7a3138dafcce0520b87b"}
Jul 28 01:34:20.016539 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.016515    2577 generic.go:356] "Generic (PLEG): container finished" podID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerID="8be9372cb349ce308121e33b942c3a940ac9d4db67962b47622632e4fdeb30c3" exitCode=0
Jul 28 01:34:20.016637 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.016590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28" event={"ID":"a68f5fb9-5801-4a5b-a256-60fd12c33817","Type":"ContainerDied","Data":"8be9372cb349ce308121e33b942c3a940ac9d4db67962b47622632e4fdeb30c3"}
Jul 28 01:34:20.146066 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.146046    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:20.240076 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.240053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-util\") pod \"b58f9679-a285-48e2-a3b2-2decdad76b1b\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") "
Jul 28 01:34:20.240244 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.240102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-bundle\") pod \"b58f9679-a285-48e2-a3b2-2decdad76b1b\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") "
Jul 28 01:34:20.240244 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.240133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m4dr\" (UniqueName: \"kubernetes.io/projected/b58f9679-a285-48e2-a3b2-2decdad76b1b-kube-api-access-8m4dr\") pod \"b58f9679-a285-48e2-a3b2-2decdad76b1b\" (UID: \"b58f9679-a285-48e2-a3b2-2decdad76b1b\") "
Jul 28 01:34:20.240758 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.240730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-bundle" (OuterVolumeSpecName: "bundle") pod "b58f9679-a285-48e2-a3b2-2decdad76b1b" (UID: "b58f9679-a285-48e2-a3b2-2decdad76b1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:20.242338 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.242314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58f9679-a285-48e2-a3b2-2decdad76b1b-kube-api-access-8m4dr" (OuterVolumeSpecName: "kube-api-access-8m4dr") pod "b58f9679-a285-48e2-a3b2-2decdad76b1b" (UID: "b58f9679-a285-48e2-a3b2-2decdad76b1b"). InnerVolumeSpecName "kube-api-access-8m4dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:34:20.245316 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.245293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-util" (OuterVolumeSpecName: "util") pod "b58f9679-a285-48e2-a3b2-2decdad76b1b" (UID: "b58f9679-a285-48e2-a3b2-2decdad76b1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:20.341135 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.341109    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:20.341135 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.341132    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58f9679-a285-48e2-a3b2-2decdad76b1b-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:20.341297 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:20.341142    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m4dr\" (UniqueName: \"kubernetes.io/projected/b58f9679-a285-48e2-a3b2-2decdad76b1b-kube-api-access-8m4dr\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.024302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.024270    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss"
Jul 28 01:34:21.024302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.024294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053hdjss" event={"ID":"b58f9679-a285-48e2-a3b2-2decdad76b1b","Type":"ContainerDied","Data":"458f2204d5da2883a4bb0e3d928671451f736a14ff1f9badd85b00914aa2e4bd"}
Jul 28 01:34:21.024882 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.024335    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="458f2204d5da2883a4bb0e3d928671451f736a14ff1f9badd85b00914aa2e4bd"
Jul 28 01:34:21.178495 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.178469    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:21.209726 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.209704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:21.213733 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.213715    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:21.248527 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-bundle\") pod \"eaf992f3-59df-421b-935f-7a08c41ce467\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") "
Jul 28 01:34:21.248673 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbc25\" (UniqueName: \"kubernetes.io/projected/eaf992f3-59df-421b-935f-7a08c41ce467-kube-api-access-kbc25\") pod \"eaf992f3-59df-421b-935f-7a08c41ce467\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") "
Jul 28 01:34:21.248673 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-bundle\") pod \"a68f5fb9-5801-4a5b-a256-60fd12c33817\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") "
Jul 28 01:34:21.248673 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-util\") pod \"a68f5fb9-5801-4a5b-a256-60fd12c33817\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") "
Jul 28 01:34:21.248673 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-util\") pod \"eaf992f3-59df-421b-935f-7a08c41ce467\" (UID: \"eaf992f3-59df-421b-935f-7a08c41ce467\") "
Jul 28 01:34:21.248911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8wxf\" (UniqueName: \"kubernetes.io/projected/a68f5fb9-5801-4a5b-a256-60fd12c33817-kube-api-access-f8wxf\") pod \"a68f5fb9-5801-4a5b-a256-60fd12c33817\" (UID: \"a68f5fb9-5801-4a5b-a256-60fd12c33817\") "
Jul 28 01:34:21.248911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-bundle\") pod \"e9713c38-ad44-419e-a12b-18839cc5401f\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") "
Jul 28 01:34:21.248911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-util\") pod \"e9713c38-ad44-419e-a12b-18839cc5401f\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") "
Jul 28 01:34:21.248911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.248831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wms5\" (UniqueName: \"kubernetes.io/projected/e9713c38-ad44-419e-a12b-18839cc5401f-kube-api-access-8wms5\") pod \"e9713c38-ad44-419e-a12b-18839cc5401f\" (UID: \"e9713c38-ad44-419e-a12b-18839cc5401f\") "
Jul 28 01:34:21.249325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.249274    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-bundle" (OuterVolumeSpecName: "bundle") pod "a68f5fb9-5801-4a5b-a256-60fd12c33817" (UID: "a68f5fb9-5801-4a5b-a256-60fd12c33817"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:21.251383 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.249406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-bundle" (OuterVolumeSpecName: "bundle") pod "eaf992f3-59df-421b-935f-7a08c41ce467" (UID: "eaf992f3-59df-421b-935f-7a08c41ce467"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:21.251383 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.250002    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-bundle" (OuterVolumeSpecName: "bundle") pod "e9713c38-ad44-419e-a12b-18839cc5401f" (UID: "e9713c38-ad44-419e-a12b-18839cc5401f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:21.251383 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.251250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf992f3-59df-421b-935f-7a08c41ce467-kube-api-access-kbc25" (OuterVolumeSpecName: "kube-api-access-kbc25") pod "eaf992f3-59df-421b-935f-7a08c41ce467" (UID: "eaf992f3-59df-421b-935f-7a08c41ce467"). InnerVolumeSpecName "kube-api-access-kbc25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:34:21.252169 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.251893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9713c38-ad44-419e-a12b-18839cc5401f-kube-api-access-8wms5" (OuterVolumeSpecName: "kube-api-access-8wms5") pod "e9713c38-ad44-419e-a12b-18839cc5401f" (UID: "e9713c38-ad44-419e-a12b-18839cc5401f"). InnerVolumeSpecName "kube-api-access-8wms5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:34:21.253765 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.253741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68f5fb9-5801-4a5b-a256-60fd12c33817-kube-api-access-f8wxf" (OuterVolumeSpecName: "kube-api-access-f8wxf") pod "a68f5fb9-5801-4a5b-a256-60fd12c33817" (UID: "a68f5fb9-5801-4a5b-a256-60fd12c33817"). InnerVolumeSpecName "kube-api-access-f8wxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:34:21.258120 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.258096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-util" (OuterVolumeSpecName: "util") pod "a68f5fb9-5801-4a5b-a256-60fd12c33817" (UID: "a68f5fb9-5801-4a5b-a256-60fd12c33817"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:21.258679 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.258659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-util" (OuterVolumeSpecName: "util") pod "eaf992f3-59df-421b-935f-7a08c41ce467" (UID: "eaf992f3-59df-421b-935f-7a08c41ce467"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:21.260008 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.259982    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-util" (OuterVolumeSpecName: "util") pod "e9713c38-ad44-419e-a12b-18839cc5401f" (UID: "e9713c38-ad44-419e-a12b-18839cc5401f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350438    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350465    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9713c38-ad44-419e-a12b-18839cc5401f-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350475    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wms5\" (UniqueName: \"kubernetes.io/projected/e9713c38-ad44-419e-a12b-18839cc5401f-kube-api-access-8wms5\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350486    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350495    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbc25\" (UniqueName: \"kubernetes.io/projected/eaf992f3-59df-421b-935f-7a08c41ce467-kube-api-access-kbc25\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350503    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-bundle\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350511    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a68f5fb9-5801-4a5b-a256-60fd12c33817-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350516 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350518    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaf992f3-59df-421b-935f-7a08c41ce467-util\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:21.350953 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:21.350525    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wxf\" (UniqueName: \"kubernetes.io/projected/a68f5fb9-5801-4a5b-a256-60fd12c33817-kube-api-access-f8wxf\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:34:22.031455 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.031429    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt"
Jul 28 01:34:22.031896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.031427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb2d5wt" event={"ID":"e9713c38-ad44-419e-a12b-18839cc5401f","Type":"ContainerDied","Data":"cd09048894431660373f91b56305451bd475e5aea31ad67821598adee16ab2f7"}
Jul 28 01:34:22.031896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.031543    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd09048894431660373f91b56305451bd475e5aea31ad67821598adee16ab2f7"
Jul 28 01:34:22.033566 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.033535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m" event={"ID":"eaf992f3-59df-421b-935f-7a08c41ce467","Type":"ContainerDied","Data":"41f534b3f3bdd9bad4f90b37c93b51dc7225c47f1ccef2f6917cb1d642771b32"}
Jul 28 01:34:22.033696 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.033575    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f534b3f3bdd9bad4f90b37c93b51dc7225c47f1ccef2f6917cb1d642771b32"
Jul 28 01:34:22.033696 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.033628    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g8f6m"
Jul 28 01:34:22.035633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.035614    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28"
Jul 28 01:34:22.035754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.035639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccqn28" event={"ID":"a68f5fb9-5801-4a5b-a256-60fd12c33817","Type":"ContainerDied","Data":"1c4b4fd2d94756363049e5a16e8e42da60d9911e456779eddac0fe70ff6dfc4b"}
Jul 28 01:34:22.035754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:22.035676    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4b4fd2d94756363049e5a16e8e42da60d9911e456779eddac0fe70ff6dfc4b"
Jul 28 01:34:26.908799 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.908766    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"]
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909126    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="extract"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909142    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="extract"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909161    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="extract"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909170    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="extract"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909187    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="util"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909197    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="util"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909208    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="pull"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909217    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="pull"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909226    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="pull"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="pull"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909244    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="pull"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909252    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="pull"
Jul 28 01:34:26.909259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909263    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909289    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="pull"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909297    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="pull"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909314    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="util"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909323    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="util"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="util"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909346    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="util"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909358    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="util"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909366    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="util"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909375    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909384    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909474    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9713c38-ad44-419e-a12b-18839cc5401f" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909485    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68f5fb9-5801-4a5b-a256-60fd12c33817" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909496    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b58f9679-a285-48e2-a3b2-2decdad76b1b" containerName="extract"
Jul 28 01:34:26.909952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.909508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf992f3-59df-421b-935f-7a08c41ce467" containerName="extract"
Jul 28 01:34:26.913842 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.913822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:26.918120 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.918095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 01:34:26.918267 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.918157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lm6dk\""
Jul 28 01:34:26.918321 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.918266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 01:34:26.918847 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.918830    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 01:34:26.944490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:26.944469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"]
Jul 28 01:34:27.001375 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:27.001348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtq9\" (UniqueName: \"kubernetes.io/projected/2e1ccc8a-070d-43e4-98ff-745e0738c01b-kube-api-access-ndtq9\") pod \"dns-operator-controller-manager-ccbdf686-5k7td\" (UID: \"2e1ccc8a-070d-43e4-98ff-745e0738c01b\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:27.102657 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:27.102630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtq9\" (UniqueName: \"kubernetes.io/projected/2e1ccc8a-070d-43e4-98ff-745e0738c01b-kube-api-access-ndtq9\") pod \"dns-operator-controller-manager-ccbdf686-5k7td\" (UID: \"2e1ccc8a-070d-43e4-98ff-745e0738c01b\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:27.112666 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:27.112630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtq9\" (UniqueName: \"kubernetes.io/projected/2e1ccc8a-070d-43e4-98ff-745e0738c01b-kube-api-access-ndtq9\") pod \"dns-operator-controller-manager-ccbdf686-5k7td\" (UID: \"2e1ccc8a-070d-43e4-98ff-745e0738c01b\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:27.223597 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:27.223574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:27.357253 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:27.357228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"]
Jul 28 01:34:27.359773 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:27.359742    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1ccc8a_070d_43e4_98ff_745e0738c01b.slice/crio-f8a4d1a3008f17ed26d9dfcf31aa05d8c39349f58b0c9890382a05fb89236e06 WatchSource:0}: Error finding container f8a4d1a3008f17ed26d9dfcf31aa05d8c39349f58b0c9890382a05fb89236e06: Status 404 returned error can't find the container with id f8a4d1a3008f17ed26d9dfcf31aa05d8c39349f58b0c9890382a05fb89236e06
Jul 28 01:34:28.067122 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:28.067086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td" event={"ID":"2e1ccc8a-070d-43e4-98ff-745e0738c01b","Type":"ContainerStarted","Data":"f8a4d1a3008f17ed26d9dfcf31aa05d8c39349f58b0c9890382a05fb89236e06"}
Jul 28 01:34:29.640392 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.640260    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-vrqn9"]
Jul 28 01:34:29.651323 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.651296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:29.656488 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.656459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pxzcg\""
Jul 28 01:34:29.685950 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.685811    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-vrqn9"]
Jul 28 01:34:29.728089 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.728057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whxx\" (UniqueName: \"kubernetes.io/projected/94f786d2-667e-45a7-8c28-87cc26a49e41-kube-api-access-4whxx\") pod \"authorino-operator-56fb6f6856-vrqn9\" (UID: \"94f786d2-667e-45a7-8c28-87cc26a49e41\") " pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:29.829112 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.829071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whxx\" (UniqueName: \"kubernetes.io/projected/94f786d2-667e-45a7-8c28-87cc26a49e41-kube-api-access-4whxx\") pod \"authorino-operator-56fb6f6856-vrqn9\" (UID: \"94f786d2-667e-45a7-8c28-87cc26a49e41\") " pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:29.838594 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.838555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whxx\" (UniqueName: \"kubernetes.io/projected/94f786d2-667e-45a7-8c28-87cc26a49e41-kube-api-access-4whxx\") pod \"authorino-operator-56fb6f6856-vrqn9\" (UID: \"94f786d2-667e-45a7-8c28-87cc26a49e41\") " pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:29.965718 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:29.965639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:30.235001 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:30.234978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-vrqn9"]
Jul 28 01:34:30.237201 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:30.237175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f786d2_667e_45a7_8c28_87cc26a49e41.slice/crio-c78b0bf70d095184208b73f231275f3968f39b6808355371fbd83482d30b60ab WatchSource:0}: Error finding container c78b0bf70d095184208b73f231275f3968f39b6808355371fbd83482d30b60ab: Status 404 returned error can't find the container with id c78b0bf70d095184208b73f231275f3968f39b6808355371fbd83482d30b60ab
Jul 28 01:34:31.085944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:31.085898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td" event={"ID":"2e1ccc8a-070d-43e4-98ff-745e0738c01b","Type":"ContainerStarted","Data":"9503df8de981cc7b2d8fba4c070bc43c0e5570a33b8813bae01db02299be9c6b"}
Jul 28 01:34:31.086566 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:31.085975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:31.087230 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:31.087203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9" event={"ID":"94f786d2-667e-45a7-8c28-87cc26a49e41","Type":"ContainerStarted","Data":"c78b0bf70d095184208b73f231275f3968f39b6808355371fbd83482d30b60ab"}
Jul 28 01:34:31.111257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:31.111206    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td" podStartSLOduration=2.324052479 podStartE2EDuration="5.111190186s" totalImagesPullingTime="2.787137707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:31.108918794 +0000 UTC m=+521.943666337" watchObservedRunningTime="2026-07-28 01:34:31.111190186 +0000 UTC m=+521.945937733"
Jul 28 01:34:33.099645 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:33.099612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9" event={"ID":"94f786d2-667e-45a7-8c28-87cc26a49e41","Type":"ContainerStarted","Data":"4761b348194a551c50e17b8b745ad0bb55891fbcee835e2a3309517a5e89cba3"}
Jul 28 01:34:33.100027 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:33.099725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:33.122909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:33.122845    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9" podStartSLOduration=2.228251815 podStartE2EDuration="4.122831031s" totalImagesPullingTime="1.894579216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:33.120666712 +0000 UTC m=+523.955414245" watchObservedRunningTime="2026-07-28 01:34:33.122831031 +0000 UTC m=+523.957578576"
Jul 28 01:34:33.978751 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:33.978677    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"]
Jul 28 01:34:33.982091 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:33.982072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:33.985082 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:33.985062    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b2j7v\""
Jul 28 01:34:34.004521 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.004500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"]
Jul 28 01:34:34.068116 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.068090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2286f348-7525-45b4-9e9c-05817b697241-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-sfbvm\" (UID: \"2286f348-7525-45b4-9e9c-05817b697241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.068272 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.068143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8l9\" (UniqueName: \"kubernetes.io/projected/2286f348-7525-45b4-9e9c-05817b697241-kube-api-access-qx8l9\") pod \"kuadrant-operator-controller-manager-66866f5fcc-sfbvm\" (UID: \"2286f348-7525-45b4-9e9c-05817b697241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.169191 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.169157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2286f348-7525-45b4-9e9c-05817b697241-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-sfbvm\" (UID: \"2286f348-7525-45b4-9e9c-05817b697241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.169651 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.169208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8l9\" (UniqueName: \"kubernetes.io/projected/2286f348-7525-45b4-9e9c-05817b697241-kube-api-access-qx8l9\") pod \"kuadrant-operator-controller-manager-66866f5fcc-sfbvm\" (UID: \"2286f348-7525-45b4-9e9c-05817b697241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.169651 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.169517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2286f348-7525-45b4-9e9c-05817b697241-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-sfbvm\" (UID: \"2286f348-7525-45b4-9e9c-05817b697241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.177996 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.177970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8l9\" (UniqueName: \"kubernetes.io/projected/2286f348-7525-45b4-9e9c-05817b697241-kube-api-access-qx8l9\") pod \"kuadrant-operator-controller-manager-66866f5fcc-sfbvm\" (UID: \"2286f348-7525-45b4-9e9c-05817b697241\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.291686 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.291629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:34.471884 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:34.471827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"]
Jul 28 01:34:34.472488 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:34:34.472464    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2286f348_7525_45b4_9e9c_05817b697241.slice/crio-c2726d1525965105016a5bbf1cc867b31998bb8340b08b4c3cff8eaf9a64e538 WatchSource:0}: Error finding container c2726d1525965105016a5bbf1cc867b31998bb8340b08b4c3cff8eaf9a64e538: Status 404 returned error can't find the container with id c2726d1525965105016a5bbf1cc867b31998bb8340b08b4c3cff8eaf9a64e538
Jul 28 01:34:35.111988 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:35.111952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm" event={"ID":"2286f348-7525-45b4-9e9c-05817b697241","Type":"ContainerStarted","Data":"c2726d1525965105016a5bbf1cc867b31998bb8340b08b4c3cff8eaf9a64e538"}
Jul 28 01:34:40.144076 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:40.144039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm" event={"ID":"2286f348-7525-45b4-9e9c-05817b697241","Type":"ContainerStarted","Data":"b783c40c9c1e3e54f564f115d3167b16327443daf89d9caa077cb378bd14422a"}
Jul 28 01:34:40.144444 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:40.144181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:34:40.166774 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:40.166287    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm" podStartSLOduration=2.330122658 podStartE2EDuration="7.166268099s" totalImagesPullingTime="4.836145441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:40.164363974 +0000 UTC m=+530.999111520" watchObservedRunningTime="2026-07-28 01:34:40.166268099 +0000 UTC m=+531.001015645"
Jul 28 01:34:42.094944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:42.094852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-5k7td"
Jul 28 01:34:44.107449 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:44.107422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-vrqn9"
Jul 28 01:34:51.152440 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:34:51.152410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-sfbvm"
Jul 28 01:35:25.735639 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.735596    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:25.738517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.738495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:25.741285 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.741263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mk4n5\""
Jul 28 01:35:25.741368 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.741263    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 01:35:25.748715 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.748693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:25.815507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.815475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzv87\" (UniqueName: \"kubernetes.io/projected/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-kube-api-access-dzv87\") pod \"limitador-limitador-859c4d96d6-llp6t\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:25.815683 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.815621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-config-file\") pod \"limitador-limitador-859c4d96d6-llp6t\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:25.831853 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.831826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:25.916116 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.916088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzv87\" (UniqueName: \"kubernetes.io/projected/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-kube-api-access-dzv87\") pod \"limitador-limitador-859c4d96d6-llp6t\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:25.916280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.916165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-config-file\") pod \"limitador-limitador-859c4d96d6-llp6t\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:25.916701 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.916684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-config-file\") pod \"limitador-limitador-859c4d96d6-llp6t\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:25.925547 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:25.925523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzv87\" (UniqueName: \"kubernetes.io/projected/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-kube-api-access-dzv87\") pod \"limitador-limitador-859c4d96d6-llp6t\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:26.049889 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:26.049784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:26.193823 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:26.193784    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:26.195106 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:35:26.195083    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eea1a57_36e0_4e7a_9962_acaec90d0d0b.slice/crio-b306c56afa48e30fa825d963428dba8a18e72f1f617a7dd92fc21985c3d12b1a WatchSource:0}: Error finding container b306c56afa48e30fa825d963428dba8a18e72f1f617a7dd92fc21985c3d12b1a: Status 404 returned error can't find the container with id b306c56afa48e30fa825d963428dba8a18e72f1f617a7dd92fc21985c3d12b1a
Jul 28 01:35:26.386286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:26.386206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t" event={"ID":"8eea1a57-36e0-4e7a-9962-acaec90d0d0b","Type":"ContainerStarted","Data":"b306c56afa48e30fa825d963428dba8a18e72f1f617a7dd92fc21985c3d12b1a"}
Jul 28 01:35:28.405650 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:28.405608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t" event={"ID":"8eea1a57-36e0-4e7a-9962-acaec90d0d0b","Type":"ContainerStarted","Data":"326d2d956af6ae75a8b711b08067b7bc32a6247200ee6cbe9a27d215bcf17be0"}
Jul 28 01:35:28.406060 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:28.405704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:28.425072 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:28.425026    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t" podStartSLOduration=2.28647448 podStartE2EDuration="3.425011488s" totalImagesPullingTime="1.138537008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:28.422284064 +0000 UTC m=+579.257031707" watchObservedRunningTime="2026-07-28 01:35:28.425011488 +0000 UTC m=+579.259759033"
Jul 28 01:35:38.900355 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:38.900299    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:38.900949 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:38.900547    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t" podUID="8eea1a57-36e0-4e7a-9962-acaec90d0d0b" containerName="limitador" containerID="cri-o://326d2d956af6ae75a8b711b08067b7bc32a6247200ee6cbe9a27d215bcf17be0" gracePeriod=30
Jul 28 01:35:38.901335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:38.901171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:39.466207 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.466131    2577 generic.go:356] "Generic (PLEG): container finished" podID="8eea1a57-36e0-4e7a-9962-acaec90d0d0b" containerID="326d2d956af6ae75a8b711b08067b7bc32a6247200ee6cbe9a27d215bcf17be0" exitCode=0
Jul 28 01:35:39.466345 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.466200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t" event={"ID":"8eea1a57-36e0-4e7a-9962-acaec90d0d0b","Type":"ContainerDied","Data":"326d2d956af6ae75a8b711b08067b7bc32a6247200ee6cbe9a27d215bcf17be0"}
Jul 28 01:35:39.849080 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.849060    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:39.944774 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.944746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzv87\" (UniqueName: \"kubernetes.io/projected/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-kube-api-access-dzv87\") pod \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") "
Jul 28 01:35:39.944774 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.944778    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-config-file\") pod \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\" (UID: \"8eea1a57-36e0-4e7a-9962-acaec90d0d0b\") "
Jul 28 01:35:39.945166 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.945131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-config-file" (OuterVolumeSpecName: "config-file") pod "8eea1a57-36e0-4e7a-9962-acaec90d0d0b" (UID: "8eea1a57-36e0-4e7a-9962-acaec90d0d0b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:35:39.946911 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:39.946882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-kube-api-access-dzv87" (OuterVolumeSpecName: "kube-api-access-dzv87") pod "8eea1a57-36e0-4e7a-9962-acaec90d0d0b" (UID: "8eea1a57-36e0-4e7a-9962-acaec90d0d0b"). InnerVolumeSpecName "kube-api-access-dzv87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:35:40.045536 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.045510    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzv87\" (UniqueName: \"kubernetes.io/projected/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-kube-api-access-dzv87\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:35:40.045536 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.045533    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8eea1a57-36e0-4e7a-9962-acaec90d0d0b-config-file\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:35:40.472906 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.472881    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t"
Jul 28 01:35:40.473061 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.472849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-llp6t" event={"ID":"8eea1a57-36e0-4e7a-9962-acaec90d0d0b","Type":"ContainerDied","Data":"b306c56afa48e30fa825d963428dba8a18e72f1f617a7dd92fc21985c3d12b1a"}
Jul 28 01:35:40.473061 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.473010    2577 scope.go:117] "RemoveContainer" containerID="326d2d956af6ae75a8b711b08067b7bc32a6247200ee6cbe9a27d215bcf17be0"
Jul 28 01:35:40.498647 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.498624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:40.500606 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:40.500587    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-llp6t"]
Jul 28 01:35:41.700011 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:41.698542    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eea1a57-36e0-4e7a-9962-acaec90d0d0b" path="/var/lib/kubelet/pods/8eea1a57-36e0-4e7a-9962-acaec90d0d0b/volumes"
Jul 28 01:35:49.029987 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.029943    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-4n9wp"]
Jul 28 01:35:49.030540 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.030500    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eea1a57-36e0-4e7a-9962-acaec90d0d0b" containerName="limitador"
Jul 28 01:35:49.030540 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.030522    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eea1a57-36e0-4e7a-9962-acaec90d0d0b" containerName="limitador"
Jul 28 01:35:49.030661 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.030642    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eea1a57-36e0-4e7a-9962-acaec90d0d0b" containerName="limitador"
Jul 28 01:35:49.033655 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.033633    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.037448 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.037426    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tj9kz\""
Jul 28 01:35:49.037567 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.037483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 01:35:49.039805 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.039783    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-4n9wp"]
Jul 28 01:35:49.121033 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.121002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8511451-bfc7-4681-ad2c-4869d499a685-tls-cert\") pod \"authorino-7cf4765b85-4n9wp\" (UID: \"e8511451-bfc7-4681-ad2c-4869d499a685\") " pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.121192 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.121040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlqr\" (UniqueName: \"kubernetes.io/projected/e8511451-bfc7-4681-ad2c-4869d499a685-kube-api-access-lnlqr\") pod \"authorino-7cf4765b85-4n9wp\" (UID: \"e8511451-bfc7-4681-ad2c-4869d499a685\") " pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.222302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.222270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8511451-bfc7-4681-ad2c-4869d499a685-tls-cert\") pod \"authorino-7cf4765b85-4n9wp\" (UID: \"e8511451-bfc7-4681-ad2c-4869d499a685\") " pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.222302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.222305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlqr\" (UniqueName: \"kubernetes.io/projected/e8511451-bfc7-4681-ad2c-4869d499a685-kube-api-access-lnlqr\") pod \"authorino-7cf4765b85-4n9wp\" (UID: \"e8511451-bfc7-4681-ad2c-4869d499a685\") " pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.224708 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.224689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e8511451-bfc7-4681-ad2c-4869d499a685-tls-cert\") pod \"authorino-7cf4765b85-4n9wp\" (UID: \"e8511451-bfc7-4681-ad2c-4869d499a685\") " pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.233658 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.233632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlqr\" (UniqueName: \"kubernetes.io/projected/e8511451-bfc7-4681-ad2c-4869d499a685-kube-api-access-lnlqr\") pod \"authorino-7cf4765b85-4n9wp\" (UID: \"e8511451-bfc7-4681-ad2c-4869d499a685\") " pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.344204 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.344138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-4n9wp"
Jul 28 01:35:49.476378 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.476352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-4n9wp"]
Jul 28 01:35:49.478023 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:35:49.477995    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8511451_bfc7_4681_ad2c_4869d499a685.slice/crio-23a87249247566ba9293190c9920d6800d70e83a24ec0a792df88a2d8bf3a41f WatchSource:0}: Error finding container 23a87249247566ba9293190c9920d6800d70e83a24ec0a792df88a2d8bf3a41f: Status 404 returned error can't find the container with id 23a87249247566ba9293190c9920d6800d70e83a24ec0a792df88a2d8bf3a41f
Jul 28 01:35:49.522901 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:49.522870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-4n9wp" event={"ID":"e8511451-bfc7-4681-ad2c-4869d499a685","Type":"ContainerStarted","Data":"23a87249247566ba9293190c9920d6800d70e83a24ec0a792df88a2d8bf3a41f"}
Jul 28 01:35:51.410190 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:51.410070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:35:51.410190 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:51.410140    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:35:51.538290 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:51.538207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-4n9wp" event={"ID":"e8511451-bfc7-4681-ad2c-4869d499a685","Type":"ContainerStarted","Data":"c58e02a9f4064538aeddd8f4fcc1d4dbdfc7582b0335789a9d43aafec1d98413"}
Jul 28 01:35:51.554915 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:35:51.554871    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-4n9wp" podStartSLOduration=0.807695315 podStartE2EDuration="2.554843917s" totalImagesPullingTime="1.747148602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:51.552716192 +0000 UTC m=+602.387463739" watchObservedRunningTime="2026-07-28 01:35:51.554843917 +0000 UTC m=+602.389591511"
Jul 28 01:36:00.826279 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.826249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-5lpl7"]
Jul 28 01:36:00.826734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.826489    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7" podUID="54a42542-497f-4e37-af12-43c625455789" containerName="manager" containerID="cri-o://87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5" gracePeriod=10
Jul 28 01:36:00.850604 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.850580    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rs8w6"]
Jul 28 01:36:00.853745 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.853729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:00.862747 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.862723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rs8w6"]
Jul 28 01:36:00.921835 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.921803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4tx\" (UniqueName: \"kubernetes.io/projected/fba12f0b-a968-411a-8246-a85542a7f2f2-kube-api-access-hm4tx\") pod \"kserve-controller-manager-ff96d5959-rs8w6\" (UID: \"fba12f0b-a968-411a-8246-a85542a7f2f2\") " pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:00.921991 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:00.921844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fba12f0b-a968-411a-8246-a85542a7f2f2-cert\") pod \"kserve-controller-manager-ff96d5959-rs8w6\" (UID: \"fba12f0b-a968-411a-8246-a85542a7f2f2\") " pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:01.022326 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.022298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fba12f0b-a968-411a-8246-a85542a7f2f2-cert\") pod \"kserve-controller-manager-ff96d5959-rs8w6\" (UID: \"fba12f0b-a968-411a-8246-a85542a7f2f2\") " pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:01.022469 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.022424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4tx\" (UniqueName: \"kubernetes.io/projected/fba12f0b-a968-411a-8246-a85542a7f2f2-kube-api-access-hm4tx\") pod \"kserve-controller-manager-ff96d5959-rs8w6\" (UID: \"fba12f0b-a968-411a-8246-a85542a7f2f2\") " pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:01.024586 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.024549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fba12f0b-a968-411a-8246-a85542a7f2f2-cert\") pod \"kserve-controller-manager-ff96d5959-rs8w6\" (UID: \"fba12f0b-a968-411a-8246-a85542a7f2f2\") " pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:01.031661 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.031640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4tx\" (UniqueName: \"kubernetes.io/projected/fba12f0b-a968-411a-8246-a85542a7f2f2-kube-api-access-hm4tx\") pod \"kserve-controller-manager-ff96d5959-rs8w6\" (UID: \"fba12f0b-a968-411a-8246-a85542a7f2f2\") " pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:01.082633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.082583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:36:01.123147 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.123118    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q2zl\" (UniqueName: \"kubernetes.io/projected/54a42542-497f-4e37-af12-43c625455789-kube-api-access-5q2zl\") pod \"54a42542-497f-4e37-af12-43c625455789\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") "
Jul 28 01:36:01.123306 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.123224    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert\") pod \"54a42542-497f-4e37-af12-43c625455789\" (UID: \"54a42542-497f-4e37-af12-43c625455789\") "
Jul 28 01:36:01.125344 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.125320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert" (OuterVolumeSpecName: "cert") pod "54a42542-497f-4e37-af12-43c625455789" (UID: "54a42542-497f-4e37-af12-43c625455789"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:36:01.125449 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.125429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a42542-497f-4e37-af12-43c625455789-kube-api-access-5q2zl" (OuterVolumeSpecName: "kube-api-access-5q2zl") pod "54a42542-497f-4e37-af12-43c625455789" (UID: "54a42542-497f-4e37-af12-43c625455789"). InnerVolumeSpecName "kube-api-access-5q2zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:01.213009 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.212977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:01.224108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.224013    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2zl\" (UniqueName: \"kubernetes.io/projected/54a42542-497f-4e37-af12-43c625455789-kube-api-access-5q2zl\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:36:01.224108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.224032    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54a42542-497f-4e37-af12-43c625455789-cert\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:36:01.339799 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.339772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-rs8w6"]
Jul 28 01:36:01.341743 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:36:01.341711    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba12f0b_a968_411a_8246_a85542a7f2f2.slice/crio-ff9527266f5f009eb5c9131b083c773f00e91323e44d5c0cbfb54658f1fe9afa WatchSource:0}: Error finding container ff9527266f5f009eb5c9131b083c773f00e91323e44d5c0cbfb54658f1fe9afa: Status 404 returned error can't find the container with id ff9527266f5f009eb5c9131b083c773f00e91323e44d5c0cbfb54658f1fe9afa
Jul 28 01:36:01.343057 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.343037    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:36:01.593818 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.593738    2577 generic.go:356] "Generic (PLEG): container finished" podID="54a42542-497f-4e37-af12-43c625455789" containerID="87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5" exitCode=0
Jul 28 01:36:01.593818 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.593785    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7"
Jul 28 01:36:01.594070 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.593823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7" event={"ID":"54a42542-497f-4e37-af12-43c625455789","Type":"ContainerDied","Data":"87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5"}
Jul 28 01:36:01.594070 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.593898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-5lpl7" event={"ID":"54a42542-497f-4e37-af12-43c625455789","Type":"ContainerDied","Data":"d97f26de41ba9a9dbeac2005277fc7d39ceb61e376d754117ec4752c7de327f0"}
Jul 28 01:36:01.594070 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.593933    2577 scope.go:117] "RemoveContainer" containerID="87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5"
Jul 28 01:36:01.595164 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.595142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6" event={"ID":"fba12f0b-a968-411a-8246-a85542a7f2f2","Type":"ContainerStarted","Data":"ff9527266f5f009eb5c9131b083c773f00e91323e44d5c0cbfb54658f1fe9afa"}
Jul 28 01:36:01.603353 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.603338    2577 scope.go:117] "RemoveContainer" containerID="87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5"
Jul 28 01:36:01.603618 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:36:01.603599    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5\": container with ID starting with 87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5 not found: ID does not exist" containerID="87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5"
Jul 28 01:36:01.603659 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.603628    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5"} err="failed to get container status \"87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5\": rpc error: code = NotFound desc = could not find container \"87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5\": container with ID starting with 87324e99bbc9fbc675416d416f7601b803bbd064f4b5d0c7fc53012de5d66ef5 not found: ID does not exist"
Jul 28 01:36:01.616479 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.616434    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-5lpl7"]
Jul 28 01:36:01.618969 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.618947    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-5lpl7"]
Jul 28 01:36:01.697909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:01.697851    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54a42542-497f-4e37-af12-43c625455789" path="/var/lib/kubelet/pods/54a42542-497f-4e37-af12-43c625455789/volumes"
Jul 28 01:36:02.602499 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:02.602463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6" event={"ID":"fba12f0b-a968-411a-8246-a85542a7f2f2","Type":"ContainerStarted","Data":"f3cb604cb0ffa5fb16e4843665a7463ef65112068a1ea6349d7e20271259514e"}
Jul 28 01:36:02.602927 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:02.602556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:02.620765 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:02.620713    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6" podStartSLOduration=1.9957856870000001 podStartE2EDuration="2.620696973s" totalImagesPullingTime="624.911286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:02.617980819 +0000 UTC m=+613.452728379" watchObservedRunningTime="2026-07-28 01:36:02.620696973 +0000 UTC m=+613.455444543"
Jul 28 01:36:33.613561 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:33.613479    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-rs8w6"
Jul 28 01:36:34.656074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.656038    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dj9c7"]
Jul 28 01:36:34.656636 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.656617    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54a42542-497f-4e37-af12-43c625455789" containerName="manager"
Jul 28 01:36:34.656731 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.656638    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="54a42542-497f-4e37-af12-43c625455789" containerName="manager"
Jul 28 01:36:34.656771 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.656746    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="54a42542-497f-4e37-af12-43c625455789" containerName="manager"
Jul 28 01:36:34.662137 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.662117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:34.665222 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.665101    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tlzdj\""
Jul 28 01:36:34.665345 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.665250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 01:36:34.672024 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.672001    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-b5sgr"]
Jul 28 01:36:34.676298 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.676280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dj9c7"]
Jul 28 01:36:34.676416 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.676385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:34.679244 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.679225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 01:36:34.679399 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.679380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-rpjxd\""
Jul 28 01:36:34.686690 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.686668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-b5sgr"]
Jul 28 01:36:34.716335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.716313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85edf2f2-b337-454c-b394-f364b2dfa2de-tls-certs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:34.716459 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.716379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2fs\" (UniqueName: \"kubernetes.io/projected/85edf2f2-b337-454c-b394-f364b2dfa2de-kube-api-access-rc2fs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:34.716459 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.716399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-cert\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:34.716646 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.716623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xb6\" (UniqueName: \"kubernetes.io/projected/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-kube-api-access-x6xb6\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:34.817325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.817297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85edf2f2-b337-454c-b394-f364b2dfa2de-tls-certs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:34.817469 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.817366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2fs\" (UniqueName: \"kubernetes.io/projected/85edf2f2-b337-454c-b394-f364b2dfa2de-kube-api-access-rc2fs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:34.817469 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.817394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-cert\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:34.817469 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.817418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xb6\" (UniqueName: \"kubernetes.io/projected/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-kube-api-access-x6xb6\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:34.817469 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:36:34.817457    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 01:36:34.817674 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:36:34.817543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85edf2f2-b337-454c-b394-f364b2dfa2de-tls-certs podName:85edf2f2-b337-454c-b394-f364b2dfa2de nodeName:}" failed. No retries permitted until 2026-07-28 01:36:35.317522729 +0000 UTC m=+646.152270267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/85edf2f2-b337-454c-b394-f364b2dfa2de-tls-certs") pod "model-serving-api-7b954ffc95-dj9c7" (UID: "85edf2f2-b337-454c-b394-f364b2dfa2de") : secret "model-serving-api-tls" not found
Jul 28 01:36:34.817674 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:36:34.817543    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 01:36:34.817674 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:36:34.817600    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-cert podName:5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4 nodeName:}" failed. No retries permitted until 2026-07-28 01:36:35.317583112 +0000 UTC m=+646.152330640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-cert") pod "odh-model-controller-75d9cdbddf-b5sgr" (UID: "5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4") : secret "odh-model-controller-webhook-cert" not found
Jul 28 01:36:34.829725 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.829697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xb6\" (UniqueName: \"kubernetes.io/projected/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-kube-api-access-x6xb6\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:34.829836 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:34.829824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2fs\" (UniqueName: \"kubernetes.io/projected/85edf2f2-b337-454c-b394-f364b2dfa2de-kube-api-access-rc2fs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:35.322100 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.322070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-cert\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:35.322297 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.322138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85edf2f2-b337-454c-b394-f364b2dfa2de-tls-certs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:35.324527 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.324504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85edf2f2-b337-454c-b394-f364b2dfa2de-tls-certs\") pod \"model-serving-api-7b954ffc95-dj9c7\" (UID: \"85edf2f2-b337-454c-b394-f364b2dfa2de\") " pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:35.324641 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.324504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4-cert\") pod \"odh-model-controller-75d9cdbddf-b5sgr\" (UID: \"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4\") " pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:35.575419 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.575335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:35.587798 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.587778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:35.721884 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.721834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dj9c7"]
Jul 28 01:36:35.723720 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:36:35.723685    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85edf2f2_b337_454c_b394_f364b2dfa2de.slice/crio-edfc2b1733ef4cec585b564c41f235285c57a43b7eec790d703a8fe9f704a598 WatchSource:0}: Error finding container edfc2b1733ef4cec585b564c41f235285c57a43b7eec790d703a8fe9f704a598: Status 404 returned error can't find the container with id edfc2b1733ef4cec585b564c41f235285c57a43b7eec790d703a8fe9f704a598
Jul 28 01:36:35.754711 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.754687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-b5sgr"]
Jul 28 01:36:35.755602 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:36:35.755583    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0541bf_1ff2_40e7_b7f8_0e1946f6f6b4.slice/crio-901f7035c3f6306fdf0323c9bf91fa726b57d67715c7a5d5df82cc92e1aa84ea WatchSource:0}: Error finding container 901f7035c3f6306fdf0323c9bf91fa726b57d67715c7a5d5df82cc92e1aa84ea: Status 404 returned error can't find the container with id 901f7035c3f6306fdf0323c9bf91fa726b57d67715c7a5d5df82cc92e1aa84ea
Jul 28 01:36:35.772764 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.772736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dj9c7" event={"ID":"85edf2f2-b337-454c-b394-f364b2dfa2de","Type":"ContainerStarted","Data":"edfc2b1733ef4cec585b564c41f235285c57a43b7eec790d703a8fe9f704a598"}
Jul 28 01:36:35.773786 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:35.773765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr" event={"ID":"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4","Type":"ContainerStarted","Data":"901f7035c3f6306fdf0323c9bf91fa726b57d67715c7a5d5df82cc92e1aa84ea"}
Jul 28 01:36:39.800373 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:39.800334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dj9c7" event={"ID":"85edf2f2-b337-454c-b394-f364b2dfa2de","Type":"ContainerStarted","Data":"af07317aa941b05909a1c072bd9184c86e856acab3880a30d9d7ddd25dea8051"}
Jul 28 01:36:39.800834 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:39.800401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:39.801718 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:39.801697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr" event={"ID":"5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4","Type":"ContainerStarted","Data":"2a19ef150597611d70951ac6134c8a29a0b9ceb8f4e7981b0b716dffe731c8ee"}
Jul 28 01:36:39.801775 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:39.801753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:39.817424 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:39.817384    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dj9c7" podStartSLOduration=2.831107802 podStartE2EDuration="5.81737109s" totalImagesPullingTime="2.986263288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:39.815220027 +0000 UTC m=+650.649967573" watchObservedRunningTime="2026-07-28 01:36:39.81737109 +0000 UTC m=+650.652118635"
Jul 28 01:36:39.832935 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:39.832840    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr" podStartSLOduration=2.834301752 podStartE2EDuration="5.832824651s" totalImagesPullingTime="2.998522899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:39.832097575 +0000 UTC m=+650.666845122" watchObservedRunningTime="2026-07-28 01:36:39.832824651 +0000 UTC m=+650.667572196"
Jul 28 01:36:50.810367 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:50.810336    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-b5sgr"
Jul 28 01:36:50.812204 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:50.812187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dj9c7"
Jul 28 01:36:51.632538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.632506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-twkpm"]
Jul 28 01:36:51.637424 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.637409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.640609 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.640591    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 01:36:51.641012 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.640989    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dp2vv\""
Jul 28 01:36:51.645967 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.645947    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-twkpm"]
Jul 28 01:36:51.769359 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.769322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc7c1b5-7021-42d4-a3fa-49f39a005ead-model-cache\") pod \"s3-init-twkpm\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") " pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.769525 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.769376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2bs\" (UniqueName: \"kubernetes.io/projected/abc7c1b5-7021-42d4-a3fa-49f39a005ead-kube-api-access-kw2bs\") pod \"s3-init-twkpm\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") " pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.869773 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.869740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc7c1b5-7021-42d4-a3fa-49f39a005ead-model-cache\") pod \"s3-init-twkpm\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") " pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.869773 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.869776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2bs\" (UniqueName: \"kubernetes.io/projected/abc7c1b5-7021-42d4-a3fa-49f39a005ead-kube-api-access-kw2bs\") pod \"s3-init-twkpm\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") " pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.870228 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.870100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc7c1b5-7021-42d4-a3fa-49f39a005ead-model-cache\") pod \"s3-init-twkpm\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") " pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.878244 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.878211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2bs\" (UniqueName: \"kubernetes.io/projected/abc7c1b5-7021-42d4-a3fa-49f39a005ead-kube-api-access-kw2bs\") pod \"s3-init-twkpm\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") " pod="kserve/s3-init-twkpm"
Jul 28 01:36:51.948004 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:51.947952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-twkpm"
Jul 28 01:36:52.078046 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:52.078021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-twkpm"]
Jul 28 01:36:52.079697 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:36:52.079672    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc7c1b5_7021_42d4_a3fa_49f39a005ead.slice/crio-446165967f8bcaf2ada00ccbe35afb191c8b0d65e81bf1240f859fa11cc7ac21 WatchSource:0}: Error finding container 446165967f8bcaf2ada00ccbe35afb191c8b0d65e81bf1240f859fa11cc7ac21: Status 404 returned error can't find the container with id 446165967f8bcaf2ada00ccbe35afb191c8b0d65e81bf1240f859fa11cc7ac21
Jul 28 01:36:52.874770 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:52.874728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-twkpm" event={"ID":"abc7c1b5-7021-42d4-a3fa-49f39a005ead","Type":"ContainerStarted","Data":"446165967f8bcaf2ada00ccbe35afb191c8b0d65e81bf1240f859fa11cc7ac21"}
Jul 28 01:36:55.892132 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:55.892094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-twkpm" event={"ID":"abc7c1b5-7021-42d4-a3fa-49f39a005ead","Type":"ContainerStarted","Data":"17bd63326fbe71cade8f6170b34c1f818d88fa9fd6ca8e8f8f5e44ebacbc4a92"}
Jul 28 01:36:57.908224 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:57.908189    2577 generic.go:356] "Generic (PLEG): container finished" podID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerID="17bd63326fbe71cade8f6170b34c1f818d88fa9fd6ca8e8f8f5e44ebacbc4a92" exitCode=0
Jul 28 01:36:57.908614 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:36:57.908254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-twkpm" event={"ID":"abc7c1b5-7021-42d4-a3fa-49f39a005ead","Type":"ContainerDied","Data":"17bd63326fbe71cade8f6170b34c1f818d88fa9fd6ca8e8f8f5e44ebacbc4a92"}
Jul 28 01:37:02.943237 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:37:02.943145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-twkpm" event={"ID":"abc7c1b5-7021-42d4-a3fa-49f39a005ead","Type":"ContainerStarted","Data":"5e7295639e088ae06604f8b49f47ca600972f3bb2d7f981f0b6f5b0df9e935f0"}
Jul 28 01:37:02.961594 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:37:02.961547    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-twkpm" podStartSLOduration=2.52784508 podStartE2EDuration="11.961533569s" totalImagesPullingTime="7.726578408s" totalInitContainerRuntime="1.707110081s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:36:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:02.958632385 +0000 UTC m=+673.793379931" watchObservedRunningTime="2026-07-28 01:37:02.961533569 +0000 UTC m=+673.796281115"
Jul 28 01:38:08.297989 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:08.297956    2577 generic.go:356] "Generic (PLEG): container finished" podID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerID="5e7295639e088ae06604f8b49f47ca600972f3bb2d7f981f0b6f5b0df9e935f0" exitCode=0
Jul 28 01:38:08.298372 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:08.298027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-twkpm" event={"ID":"abc7c1b5-7021-42d4-a3fa-49f39a005ead","Type":"ContainerDied","Data":"5e7295639e088ae06604f8b49f47ca600972f3bb2d7f981f0b6f5b0df9e935f0"}
Jul 28 01:38:09.436004 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.435982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-twkpm"
Jul 28 01:38:09.540268 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.540238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw2bs\" (UniqueName: \"kubernetes.io/projected/abc7c1b5-7021-42d4-a3fa-49f39a005ead-kube-api-access-kw2bs\") pod \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") "
Jul 28 01:38:09.540453 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.540333    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc7c1b5-7021-42d4-a3fa-49f39a005ead-model-cache\") pod \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\" (UID: \"abc7c1b5-7021-42d4-a3fa-49f39a005ead\") "
Jul 28 01:38:09.542325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.542294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc7c1b5-7021-42d4-a3fa-49f39a005ead-kube-api-access-kw2bs" (OuterVolumeSpecName: "kube-api-access-kw2bs") pod "abc7c1b5-7021-42d4-a3fa-49f39a005ead" (UID: "abc7c1b5-7021-42d4-a3fa-49f39a005ead"). InnerVolumeSpecName "kube-api-access-kw2bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:38:09.558937 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.558881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc7c1b5-7021-42d4-a3fa-49f39a005ead-model-cache" (OuterVolumeSpecName: "model-cache") pod "abc7c1b5-7021-42d4-a3fa-49f39a005ead" (UID: "abc7c1b5-7021-42d4-a3fa-49f39a005ead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:38:09.641456 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.641430    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2bs\" (UniqueName: \"kubernetes.io/projected/abc7c1b5-7021-42d4-a3fa-49f39a005ead-kube-api-access-kw2bs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:38:09.641456 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:09.641452    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc7c1b5-7021-42d4-a3fa-49f39a005ead-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:38:10.317189 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:10.317147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-twkpm" event={"ID":"abc7c1b5-7021-42d4-a3fa-49f39a005ead","Type":"ContainerDied","Data":"446165967f8bcaf2ada00ccbe35afb191c8b0d65e81bf1240f859fa11cc7ac21"}
Jul 28 01:38:10.317189 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:10.317195    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446165967f8bcaf2ada00ccbe35afb191c8b0d65e81bf1240f859fa11cc7ac21"
Jul 28 01:38:10.317429 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:10.317202    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-twkpm"
Jul 28 01:38:26.859591 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.859560    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"]
Jul 28 01:38:26.860040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.859972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerName="download-hf-model"
Jul 28 01:38:26.860040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.859985    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerName="download-hf-model"
Jul 28 01:38:26.860040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.860006    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerName="s3-init"
Jul 28 01:38:26.860040 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.860011    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerName="s3-init"
Jul 28 01:38:26.860169 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.860067    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc7c1b5-7021-42d4-a3fa-49f39a005ead" containerName="s3-init"
Jul 28 01:38:26.863344 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.863325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.867517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.866448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 28 01:38:26.867517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.866787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 28 01:38:26.867517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.867060    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 28 01:38:26.867517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.867269    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 28 01:38:26.867517 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.867290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-q9sns\""
Jul 28 01:38:26.875279 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.875254    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"]
Jul 28 01:38:26.889134 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.889268 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.889268 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.889382 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.889382 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.889382 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.889382 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.889350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kube-api-access-sk56m\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990258 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990258 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990467 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kube-api-access-sk56m\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990738 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990801 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990857 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.990923 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.990886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.992647 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.992621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.992990 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.992970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:26.998153 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:26.998133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kube-api-access-sk56m\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:27.177621 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:27.177543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:38:27.360333 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:38:27.360289    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f220e8_bf8c_41d2_a0aa_a136b6c1ec74.slice/crio-b228c4ffdcf00a818ed081aa302920dda26dae31a0ed0419c3a6e3a290e5c1fa WatchSource:0}: Error finding container b228c4ffdcf00a818ed081aa302920dda26dae31a0ed0419c3a6e3a290e5c1fa: Status 404 returned error can't find the container with id b228c4ffdcf00a818ed081aa302920dda26dae31a0ed0419c3a6e3a290e5c1fa
Jul 28 01:38:27.364405 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:27.364377    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"]
Jul 28 01:38:27.412130 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:27.412099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" event={"ID":"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74","Type":"ContainerStarted","Data":"b228c4ffdcf00a818ed081aa302920dda26dae31a0ed0419c3a6e3a290e5c1fa"}
Jul 28 01:38:28.421688 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:28.421645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" event={"ID":"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74","Type":"ContainerStarted","Data":"1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b"}
Jul 28 01:38:32.450417 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:32.450339    2577 generic.go:356] "Generic (PLEG): container finished" podID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerID="1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b" exitCode=0
Jul 28 01:38:32.450740 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:38:32.450415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" event={"ID":"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74","Type":"ContainerDied","Data":"1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b"}
Jul 28 01:39:02.690330 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:02.690293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" event={"ID":"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74","Type":"ContainerStarted","Data":"62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0"}
Jul 28 01:39:02.717107 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:02.717057    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podStartSLOduration=2.6225856199999997 podStartE2EDuration="36.7170426s" totalImagesPullingTime="29.484715192s" totalInitContainerRuntime="4.609741788s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:02.714458443 +0000 UTC m=+793.549206002" watchObservedRunningTime="2026-07-28 01:39:02.7170426 +0000 UTC m=+793.551790146"
Jul 28 01:39:03.178020 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:03.177986    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:39:07.177904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:07.177852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:39:07.179477 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:07.179449    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:39:17.178486 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:17.178400    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:39:27.178711 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:27.178672    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:39:37.178468 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:37.178423    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:39:47.178413 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:47.178368    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:39:57.178419 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:39:57.178373    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:40:07.178774 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:07.178732    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:40:17.178478 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:17.178426    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:40:27.178225 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:27.178177    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:40:37.178457 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:37.178411    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:40:47.178470 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:47.178391    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:40:51.451683 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:51.451572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:40:51.453661 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:51.453629    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:40:57.188032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:57.187993    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:40:57.196144 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:40:57.196117    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:41:18.663807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:18.663775    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"]
Jul 28 01:41:33.686788 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:33.686752    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" containerID="cri-o://62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0" gracePeriod=45
Jul 28 01:41:34.189069 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.189026    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 28 01:41:34.459645 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.459623    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:41:34.547012 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.546988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-dshm\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547151 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-home\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547151 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kserve-provision-location\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547151 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547090    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tmp-dir\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tls-certs\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-model-cache\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kube-api-access-sk56m\") pod \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\" (UID: \"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74\") "
Jul 28 01:41:34.547784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-home" (OuterVolumeSpecName: "home") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:34.547784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.547726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:34.549406 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.549369    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:41:34.549528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.549426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-dshm" (OuterVolumeSpecName: "dshm") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:34.549528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.549455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kube-api-access-sk56m" (OuterVolumeSpecName: "kube-api-access-sk56m") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "kube-api-access-sk56m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:34.560048 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.560013    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:34.573529 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.573505    2577 generic.go:356] "Generic (PLEG): container finished" podID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerID="62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0" exitCode=0
Jul 28 01:41:34.573627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.573566    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"
Jul 28 01:41:34.573627 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.573590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" event={"ID":"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74","Type":"ContainerDied","Data":"62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0"}
Jul 28 01:41:34.573697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.573657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h" event={"ID":"c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74","Type":"ContainerDied","Data":"b228c4ffdcf00a818ed081aa302920dda26dae31a0ed0419c3a6e3a290e5c1fa"}
Jul 28 01:41:34.573697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.573688    2577 scope.go:117] "RemoveContainer" containerID="62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0"
Jul 28 01:41:34.582333 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.582317    2577 scope.go:117] "RemoveContainer" containerID="1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b"
Jul 28 01:41:34.590213 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.590193    2577 scope.go:117] "RemoveContainer" containerID="62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0"
Jul 28 01:41:34.590474 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:41:34.590456    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0\": container with ID starting with 62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0 not found: ID does not exist" containerID="62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0"
Jul 28 01:41:34.590534 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.590482    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0"} err="failed to get container status \"62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0\": rpc error: code = NotFound desc = could not find container \"62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0\": container with ID starting with 62b62e6d798645c23d272ed11e7c0ef2f0ee07066ec780b9ce3c188afb5db8a0 not found: ID does not exist"
Jul 28 01:41:34.590534 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.590499    2577 scope.go:117] "RemoveContainer" containerID="1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b"
Jul 28 01:41:34.590721 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:41:34.590707    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b\": container with ID starting with 1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b not found: ID does not exist" containerID="1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b"
Jul 28 01:41:34.590761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.590726    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b"} err="failed to get container status \"1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b\": rpc error: code = NotFound desc = could not find container \"1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b\": container with ID starting with 1cd9e6b55da5586ed14b5fe3de9e6284d8b325e7c69444ef76e97951d7c33f1b not found: ID does not exist"
Jul 28 01:41:34.607296 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.607273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" (UID: "c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:34.647996 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.647978    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.648056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.647996    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.648056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.648016    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kube-api-access-sk56m\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.648056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.648026    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.648056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.648035    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.648056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.648043    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.648056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.648052    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:41:34.898227 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.898201    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"]
Jul 28 01:41:34.903603 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:34.903578    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78cd7c4d8fz7z4h"]
Jul 28 01:41:35.698868 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:35.698825    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" path="/var/lib/kubelet/pods/c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74/volumes"
Jul 28 01:41:46.742547 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.742513    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"]
Jul 28 01:41:46.742956 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.742942    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="storage-initializer"
Jul 28 01:41:46.742956 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.742957    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="storage-initializer"
Jul 28 01:41:46.743038 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.742968    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main"
Jul 28 01:41:46.743038 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.742974    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main"
Jul 28 01:41:46.743038 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.743033    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f220e8-bf8c-41d2-a0aa-a136b6c1ec74" containerName="main"
Jul 28 01:41:46.746184 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.746167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.749083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.749048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 01:41:46.749208 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.749085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 28 01:41:46.749208 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.749051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 01:41:46.749208 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.749047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 01:41:46.749208 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.749154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-9xx4h\""
Jul 28 01:41:46.756419 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-dshm\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.756541 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tls-certs\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.756541 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qsm\" (UniqueName: \"kubernetes.io/projected/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kube-api-access-r4qsm\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.756670 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kserve-provision-location\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.756670 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-home\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.756777 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-model-cache\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.756777 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.756711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tmp-dir\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.757811 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.757791    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"]
Jul 28 01:41:46.857253 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tls-certs\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857253 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qsm\" (UniqueName: \"kubernetes.io/projected/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kube-api-access-r4qsm\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kserve-provision-location\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-home\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-model-cache\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tmp-dir\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857490 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-dshm\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857731 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kserve-provision-location\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857795 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-home\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857795 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-model-cache\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.857881 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.857802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tmp-dir\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.859611 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.859590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-dshm\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.859896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.859876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tls-certs\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:46.866059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:46.866038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qsm\" (UniqueName: \"kubernetes.io/projected/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kube-api-access-r4qsm\") pod \"auth-enabled-test-kserve-85d957ff8-nprdm\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:47.058181 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:47.058100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:47.223273 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:47.223250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"]
Jul 28 01:41:47.225460 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:41:47.225434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4193cdcb_719d_4cb3_bf6c_a9432aaef688.slice/crio-447002f2f9bae7ad1047317da0656dc169b403de11a06c2bae88ae3257a3216e WatchSource:0}: Error finding container 447002f2f9bae7ad1047317da0656dc169b403de11a06c2bae88ae3257a3216e: Status 404 returned error can't find the container with id 447002f2f9bae7ad1047317da0656dc169b403de11a06c2bae88ae3257a3216e
Jul 28 01:41:47.647954 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:47.647910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" event={"ID":"4193cdcb-719d-4cb3-bf6c-a9432aaef688","Type":"ContainerStarted","Data":"9a0e5327a55ba6087cd8e1f8c11e3f7352c2ee673a06a9db4682019c10b5d660"}
Jul 28 01:41:47.648146 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:47.647998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" event={"ID":"4193cdcb-719d-4cb3-bf6c-a9432aaef688","Type":"ContainerStarted","Data":"447002f2f9bae7ad1047317da0656dc169b403de11a06c2bae88ae3257a3216e"}
Jul 28 01:41:52.678580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:52.678547    2577 generic.go:356] "Generic (PLEG): container finished" podID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerID="9a0e5327a55ba6087cd8e1f8c11e3f7352c2ee673a06a9db4682019c10b5d660" exitCode=0
Jul 28 01:41:52.678580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:52.678584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" event={"ID":"4193cdcb-719d-4cb3-bf6c-a9432aaef688","Type":"ContainerDied","Data":"9a0e5327a55ba6087cd8e1f8c11e3f7352c2ee673a06a9db4682019c10b5d660"}
Jul 28 01:41:53.686650 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:53.686619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" event={"ID":"4193cdcb-719d-4cb3-bf6c-a9432aaef688","Type":"ContainerStarted","Data":"4963b2e013b61e6a41c0e4010f1731212aa3ee6f1ff48d8a1a8597de8a313acf"}
Jul 28 01:41:53.710759 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:53.710705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podStartSLOduration=2.485855022 podStartE2EDuration="7.710688055s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.224833033s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:53.708585699 +0000 UTC m=+964.543333256" watchObservedRunningTime="2026-07-28 01:41:53.710688055 +0000 UTC m=+964.545435601"
Jul 28 01:41:54.058514 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:54.058472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:56.370373 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.370334    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 01:41:56.373492 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.373476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.376392 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.376368    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 01:41:56.376496 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.376403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 01:41:56.377278 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.377254    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-2pbh9\""
Jul 28 01:41:56.377278 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.377269    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 01:41:56.377440 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.377333    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-6sntk\""
Jul 28 01:41:56.377440 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.377361    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 01:41:56.390921 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.390896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 01:41:56.436582 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.436582 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.436815 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.436815 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.436815 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.436815 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2nb\" (UniqueName: \"kubernetes.io/projected/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kube-api-access-6p2nb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.437032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.436878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2nb\" (UniqueName: \"kubernetes.io/projected/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kube-api-access-6p2nb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538626 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538755 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.538947 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.538922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.540499 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.540476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.540678 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.540662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.551714 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.551683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2nb\" (UniqueName: \"kubernetes.io/projected/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kube-api-access-6p2nb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.684544 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.684466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:41:56.846823 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:56.846799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 01:41:56.849506 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:41:56.849473    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545068dd_56d6_4c7a_bec0_a1c8181d70d8.slice/crio-1f15b55cd960ff6f32b7f74af2cfd96cb506bc482d8e81388d3e6445872d7f19 WatchSource:0}: Error finding container 1f15b55cd960ff6f32b7f74af2cfd96cb506bc482d8e81388d3e6445872d7f19: Status 404 returned error can't find the container with id 1f15b55cd960ff6f32b7f74af2cfd96cb506bc482d8e81388d3e6445872d7f19
Jul 28 01:41:57.059180 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:57.059140    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:41:57.060739 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:57.060703    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:41:57.711699 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:57.711656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"545068dd-56d6-4c7a-bec0-a1c8181d70d8","Type":"ContainerStarted","Data":"b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575"}
Jul 28 01:41:57.712152 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:41:57.711724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"545068dd-56d6-4c7a-bec0-a1c8181d70d8","Type":"ContainerStarted","Data":"1f15b55cd960ff6f32b7f74af2cfd96cb506bc482d8e81388d3e6445872d7f19"}
Jul 28 01:42:05.763280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:05.763241    2577 generic.go:356] "Generic (PLEG): container finished" podID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerID="b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575" exitCode=0
Jul 28 01:42:05.763816 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:05.763314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"545068dd-56d6-4c7a-bec0-a1c8181d70d8","Type":"ContainerDied","Data":"b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575"}
Jul 28 01:42:06.773717 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:06.773671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"545068dd-56d6-4c7a-bec0-a1c8181d70d8","Type":"ContainerStarted","Data":"7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef"}
Jul 28 01:42:06.797635 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:06.797572    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.709193482 podStartE2EDuration="10.797554791s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.088361309s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:42:06.79530321 +0000 UTC m=+977.630050756" watchObservedRunningTime="2026-07-28 01:42:06.797554791 +0000 UTC m=+977.632302340"
Jul 28 01:42:07.059625 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:07.059526    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:42:17.059634 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:17.059544    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:42:27.059025 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:27.058974    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:42:37.059406 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:37.059352    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:42:47.058907 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:47.058839    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:42:57.058447 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:42:57.058400    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:43:07.059377 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:43:07.059339    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:43:17.059263 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:43:17.059213    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:43:27.059407 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:43:27.059361    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:43:37.059462 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:43:37.059407    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:43:47.059021 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:43:47.058924    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:43:57.058668 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:43:57.058625    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:44:07.059452 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:07.059412    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:44:17.058434 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:17.058391    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 28 01:44:27.069090 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:27.069053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:44:27.076700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:27.076680    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:44:32.095940 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:32.095907    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"]
Jul 28 01:44:47.121132 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.121065    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main" containerID="cri-o://4963b2e013b61e6a41c0e4010f1731212aa3ee6f1ff48d8a1a8597de8a313acf" gracePeriod=45
Jul 28 01:44:47.765081 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.765047    2577 generic.go:356] "Generic (PLEG): container finished" podID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerID="4963b2e013b61e6a41c0e4010f1731212aa3ee6f1ff48d8a1a8597de8a313acf" exitCode=0
Jul 28 01:44:47.765252 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.765123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" event={"ID":"4193cdcb-719d-4cb3-bf6c-a9432aaef688","Type":"ContainerDied","Data":"4963b2e013b61e6a41c0e4010f1731212aa3ee6f1ff48d8a1a8597de8a313acf"}
Jul 28 01:44:47.882561 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.882499    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:44:47.992847 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.992818    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kserve-provision-location\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993025 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.992923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tls-certs\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993025 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.992942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-model-cache\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993025 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.992957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tmp-dir\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993025 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.992985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4qsm\" (UniqueName: \"kubernetes.io/projected/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kube-api-access-r4qsm\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993025 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.993016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-dshm\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.993031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-home\") pod \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\" (UID: \"4193cdcb-719d-4cb3-bf6c-a9432aaef688\") "
Jul 28 01:44:47.993302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.993220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-model-cache" (OuterVolumeSpecName: "model-cache") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:47.993976 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.993949    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-home" (OuterVolumeSpecName: "home") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:47.995387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.995361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:44:47.995387 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.995375    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kube-api-access-r4qsm" (OuterVolumeSpecName: "kube-api-access-r4qsm") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "kube-api-access-r4qsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:44:47.995549 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:47.995418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-dshm" (OuterVolumeSpecName: "dshm") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:48.005673 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.005646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:48.056120 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.056083    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4193cdcb-719d-4cb3-bf6c-a9432aaef688" (UID: "4193cdcb-719d-4cb3-bf6c-a9432aaef688"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:48.093924 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093893    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.093924 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093923    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.094055 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093931    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.094055 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093942    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4qsm\" (UniqueName: \"kubernetes.io/projected/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kube-api-access-r4qsm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.094055 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093953    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.094055 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093966    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.094055 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.093977    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4193cdcb-719d-4cb3-bf6c-a9432aaef688-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:44:48.772265 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.772232    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"
Jul 28 01:44:48.772700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.772265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm" event={"ID":"4193cdcb-719d-4cb3-bf6c-a9432aaef688","Type":"ContainerDied","Data":"447002f2f9bae7ad1047317da0656dc169b403de11a06c2bae88ae3257a3216e"}
Jul 28 01:44:48.772700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.772327    2577 scope.go:117] "RemoveContainer" containerID="4963b2e013b61e6a41c0e4010f1731212aa3ee6f1ff48d8a1a8597de8a313acf"
Jul 28 01:44:48.784979 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.784955    2577 scope.go:117] "RemoveContainer" containerID="9a0e5327a55ba6087cd8e1f8c11e3f7352c2ee673a06a9db4682019c10b5d660"
Jul 28 01:44:48.796220 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.796196    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"]
Jul 28 01:44:48.799946 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:48.799924    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-nprdm"]
Jul 28 01:44:49.697825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:49.697791    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" path="/var/lib/kubelet/pods/4193cdcb-719d-4cb3-bf6c-a9432aaef688/volumes"
Jul 28 01:44:50.692469 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:50.692436    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-6sntk\" not found"
Jul 28 01:44:50.822116 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:50.822084    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:50.822282 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:50.822158    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs podName:545068dd-56d6-4c7a-bec0-a1c8181d70d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:44:51.322136916 +0000 UTC m=+1142.156884440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:51.327651 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:51.327607    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:51.327894 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:51.327708    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs podName:545068dd-56d6-4c7a-bec0-a1c8181d70d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:44:52.327689533 +0000 UTC m=+1143.162437058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:52.336505 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:52.336472    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:52.336931 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:52.336541    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs podName:545068dd-56d6-4c7a-bec0-a1c8181d70d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:44:54.336524455 +0000 UTC m=+1145.171271998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:54.356629 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:54.356593    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:54.357115 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:54.356687    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs podName:545068dd-56d6-4c7a-bec0-a1c8181d70d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:44:58.356669916 +0000 UTC m=+1149.191417453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:54.640904 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:44:54.640803    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 01:44:58.393666 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:58.393635    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:44:58.394133 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:44:58.393699    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs podName:545068dd-56d6-4c7a-bec0-a1c8181d70d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:45:06.393684313 +0000 UTC m=+1157.228431838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:45:06.472128 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:45:06.472095    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:45:06.472537 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:45:06.472174    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs podName:545068dd-56d6-4c7a-bec0-a1c8181d70d8 nodeName:}" failed. No retries permitted until 2026-07-28 01:45:22.472158295 +0000 UTC m=+1173.306905820 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 01:45:09.663463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:09.663423    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerName="main" containerID="cri-o://7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef" gracePeriod=45
Jul 28 01:45:10.326374 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.326351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:45:10.402157 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-home\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402310 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402310 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-dshm\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402310 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402233    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-model-cache\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402310 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kserve-provision-location\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402503 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p2nb\" (UniqueName: \"kubernetes.io/projected/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kube-api-access-6p2nb\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402503 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tmp-dir\") pod \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\" (UID: \"545068dd-56d6-4c7a-bec0-a1c8181d70d8\") "
Jul 28 01:45:10.402614 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:45:10.402763 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402742    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.402850 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.402834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-home" (OuterVolumeSpecName: "home") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:45:10.404525 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.404501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-dshm" (OuterVolumeSpecName: "dshm") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:45:10.404616 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.404525    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kube-api-access-6p2nb" (OuterVolumeSpecName: "kube-api-access-6p2nb") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "kube-api-access-6p2nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:45:10.404706 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.404682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:45:10.414349 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.414325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:45:10.460520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.460450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "545068dd-56d6-4c7a-bec0-a1c8181d70d8" (UID: "545068dd-56d6-4c7a-bec0-a1c8181d70d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:45:10.503240 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.503208    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p2nb\" (UniqueName: \"kubernetes.io/projected/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kube-api-access-6p2nb\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.503240 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.503237    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.503411 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.503254    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.503411 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.503263    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545068dd-56d6-4c7a-bec0-a1c8181d70d8-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.503411 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.503272    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.503411 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.503279    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545068dd-56d6-4c7a-bec0-a1c8181d70d8-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:45:10.904364 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.904325    2577 generic.go:356] "Generic (PLEG): container finished" podID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerID="7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef" exitCode=0
Jul 28 01:45:10.904780 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.904371    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 01:45:10.904780 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.904377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"545068dd-56d6-4c7a-bec0-a1c8181d70d8","Type":"ContainerDied","Data":"7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef"}
Jul 28 01:45:10.904780 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.904423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"545068dd-56d6-4c7a-bec0-a1c8181d70d8","Type":"ContainerDied","Data":"1f15b55cd960ff6f32b7f74af2cfd96cb506bc482d8e81388d3e6445872d7f19"}
Jul 28 01:45:10.904780 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.904446    2577 scope.go:117] "RemoveContainer" containerID="7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef"
Jul 28 01:45:10.917135 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.917111    2577 scope.go:117] "RemoveContainer" containerID="b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575"
Jul 28 01:45:10.926066 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.926046    2577 scope.go:117] "RemoveContainer" containerID="7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef"
Jul 28 01:45:10.926322 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:45:10.926304    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef\": container with ID starting with 7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef not found: ID does not exist" containerID="7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef"
Jul 28 01:45:10.926363 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.926333    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef"} err="failed to get container status \"7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef\": rpc error: code = NotFound desc = could not find container \"7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef\": container with ID starting with 7b3d6897bd653169e6e00a2369ac111f60d7a55163ffa3357f20da683ff915ef not found: ID does not exist"
Jul 28 01:45:10.926363 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.926349    2577 scope.go:117] "RemoveContainer" containerID="b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575"
Jul 28 01:45:10.926597 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:45:10.926579    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575\": container with ID starting with b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575 not found: ID does not exist" containerID="b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575"
Jul 28 01:45:10.926633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.926603    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575"} err="failed to get container status \"b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575\": rpc error: code = NotFound desc = could not find container \"b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575\": container with ID starting with b2d4fbb16bcbcc9d7399196339ded3e7d0a299def9bc139d8a06a12007ded575 not found: ID does not exist"
Jul 28 01:45:10.933763 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.933738    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 01:45:10.937519 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:10.937496    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 01:45:11.698307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:11.698277    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" path="/var/lib/kubelet/pods/545068dd-56d6-4c7a-bec0-a1c8181d70d8/volumes"
Jul 28 01:45:40.562279 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562245    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"]
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562616    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="storage-initializer"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562627    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="storage-initializer"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562643    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerName="storage-initializer"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562651    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerName="storage-initializer"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562667    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562674    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562687    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerName="main"
Jul 28 01:45:40.562727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerName="main"
Jul 28 01:45:40.563016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562758    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="545068dd-56d6-4c7a-bec0-a1c8181d70d8" containerName="main"
Jul 28 01:45:40.563016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.562768    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4193cdcb-719d-4cb3-bf6c-a9432aaef688" containerName="main"
Jul 28 01:45:40.565840 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.565820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.570568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.570542    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 01:45:40.570568 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.570562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 01:45:40.570745 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.570603    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 01:45:40.570745 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.570543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-t5mtz\""
Jul 28 01:45:40.570846 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.570798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-vn47b\""
Jul 28 01:45:40.584575 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.584554    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"]
Jul 28 01:45:40.677181 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.677143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6n7\" (UniqueName: \"kubernetes.io/projected/2f91e70f-53a3-4ab2-86e1-2929694b301c-kube-api-access-hx6n7\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.677341 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.677278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91e70f-53a3-4ab2-86e1-2929694b301c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.778496 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.778458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91e70f-53a3-4ab2-86e1-2929694b301c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.778660 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.778556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6n7\" (UniqueName: \"kubernetes.io/projected/2f91e70f-53a3-4ab2-86e1-2929694b301c-kube-api-access-hx6n7\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.781045 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.781015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91e70f-53a3-4ab2-86e1-2929694b301c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.789930 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.789908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6n7\" (UniqueName: \"kubernetes.io/projected/2f91e70f-53a3-4ab2-86e1-2929694b301c-kube-api-access-hx6n7\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:40.877499 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:40.877388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:41.023409 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:41.023384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"]
Jul 28 01:45:41.024567 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:45:41.024541    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f91e70f_53a3_4ab2_86e1_2929694b301c.slice/crio-75444a46682490416282a6c3a3ecda70f2b0c0795ff23e5e860798703c365456 WatchSource:0}: Error finding container 75444a46682490416282a6c3a3ecda70f2b0c0795ff23e5e860798703c365456: Status 404 returned error can't find the container with id 75444a46682490416282a6c3a3ecda70f2b0c0795ff23e5e860798703c365456
Jul 28 01:45:41.026406 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:41.026389    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:45:41.072020 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:41.071994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" event={"ID":"2f91e70f-53a3-4ab2-86e1-2929694b301c","Type":"ContainerStarted","Data":"75444a46682490416282a6c3a3ecda70f2b0c0795ff23e5e860798703c365456"}
Jul 28 01:45:44.094288 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:44.094250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" event={"ID":"2f91e70f-53a3-4ab2-86e1-2929694b301c","Type":"ContainerStarted","Data":"06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee"}
Jul 28 01:45:44.094744 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:44.094355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:45:44.112366 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:44.112318    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" podStartSLOduration=1.610671621 podStartE2EDuration="4.112303936s" totalImagesPullingTime="2.501632315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:44.110733211 +0000 UTC m=+1194.945480756" watchObservedRunningTime="2026-07-28 01:45:44.112303936 +0000 UTC m=+1194.947051481"
Jul 28 01:45:51.495031 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:51.495005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:45:51.497340 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:45:51.496756    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:46:15.103542 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:15.103511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:46:16.477051 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:16.477015    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"]
Jul 28 01:46:25.103452 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:25.103419    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" podUID="2f91e70f-53a3-4ab2-86e1-2929694b301c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:46:31.496912 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.496830    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" podUID="2f91e70f-53a3-4ab2-86e1-2929694b301c" containerName="main" containerID="cri-o://06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee" gracePeriod=45
Jul 28 01:46:31.746607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.746585    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:46:31.839436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.839352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91e70f-53a3-4ab2-86e1-2929694b301c-tls-certs\") pod \"2f91e70f-53a3-4ab2-86e1-2929694b301c\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") "
Jul 28 01:46:31.839436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.839424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx6n7\" (UniqueName: \"kubernetes.io/projected/2f91e70f-53a3-4ab2-86e1-2929694b301c-kube-api-access-hx6n7\") pod \"2f91e70f-53a3-4ab2-86e1-2929694b301c\" (UID: \"2f91e70f-53a3-4ab2-86e1-2929694b301c\") "
Jul 28 01:46:31.841680 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.841655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f91e70f-53a3-4ab2-86e1-2929694b301c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f91e70f-53a3-4ab2-86e1-2929694b301c" (UID: "2f91e70f-53a3-4ab2-86e1-2929694b301c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:46:31.841792 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.841655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f91e70f-53a3-4ab2-86e1-2929694b301c-kube-api-access-hx6n7" (OuterVolumeSpecName: "kube-api-access-hx6n7") pod "2f91e70f-53a3-4ab2-86e1-2929694b301c" (UID: "2f91e70f-53a3-4ab2-86e1-2929694b301c"). InnerVolumeSpecName "kube-api-access-hx6n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:46:31.940436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.940410    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91e70f-53a3-4ab2-86e1-2929694b301c-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:46:31.940436 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:31.940433    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6n7\" (UniqueName: \"kubernetes.io/projected/2f91e70f-53a3-4ab2-86e1-2929694b301c-kube-api-access-hx6n7\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:46:32.353094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.353053    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f91e70f-53a3-4ab2-86e1-2929694b301c" containerID="06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee" exitCode=0
Jul 28 01:46:32.353307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.353104    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"
Jul 28 01:46:32.353307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.353123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" event={"ID":"2f91e70f-53a3-4ab2-86e1-2929694b301c","Type":"ContainerDied","Data":"06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee"}
Jul 28 01:46:32.353307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.353164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj" event={"ID":"2f91e70f-53a3-4ab2-86e1-2929694b301c","Type":"ContainerDied","Data":"75444a46682490416282a6c3a3ecda70f2b0c0795ff23e5e860798703c365456"}
Jul 28 01:46:32.353307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.353186    2577 scope.go:117] "RemoveContainer" containerID="06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee"
Jul 28 01:46:32.362734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.362718    2577 scope.go:117] "RemoveContainer" containerID="06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee"
Jul 28 01:46:32.363009 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:46:32.362990    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee\": container with ID starting with 06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee not found: ID does not exist" containerID="06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee"
Jul 28 01:46:32.363069 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.363017    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee"} err="failed to get container status \"06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee\": rpc error: code = NotFound desc = could not find container \"06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee\": container with ID starting with 06fd99a4c71b5085607e5bd65751955348e9e68f1ebda0e0024e5592c9a281ee not found: ID does not exist"
Jul 28 01:46:32.377225 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.377200    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"]
Jul 28 01:46:32.382051 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:32.382031    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5lwhwj"]
Jul 28 01:46:33.697562 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:33.697526    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f91e70f-53a3-4ab2-86e1-2929694b301c" path="/var/lib/kubelet/pods/2f91e70f-53a3-4ab2-86e1-2929694b301c/volumes"
Jul 28 01:46:47.752074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.750299    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"]
Jul 28 01:46:47.752074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.750924    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f91e70f-53a3-4ab2-86e1-2929694b301c" containerName="main"
Jul 28 01:46:47.752074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.750944    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f91e70f-53a3-4ab2-86e1-2929694b301c" containerName="main"
Jul 28 01:46:47.752074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.751080    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f91e70f-53a3-4ab2-86e1-2929694b301c" containerName="main"
Jul 28 01:46:47.756481 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.756453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"]
Jul 28 01:46:47.756645 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.756557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.759324 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.759289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 01:46:47.760162 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.760128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-jf444\""
Jul 28 01:46:47.760275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.760133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 01:46:47.760986 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.760963    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 01:46:47.776779 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.776747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.776959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.776832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.776959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.776926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.777083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.776966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.777083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.776994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.777083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.777039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jqn\" (UniqueName: \"kubernetes.io/projected/1acb7079-2d25-4f03-8b17-dfc06e4bebda-kube-api-access-l2jqn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.877898 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.877827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878086 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.877905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878086 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.877942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jqn\" (UniqueName: \"kubernetes.io/projected/1acb7079-2d25-4f03-8b17-dfc06e4bebda-kube-api-access-l2jqn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878086 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.877985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878086 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.878054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878319 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.878101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878319 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.878203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878319 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.878297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.878498 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.878417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.880445 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.880421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.880639 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.880622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:47.891128 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:47.891104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jqn\" (UniqueName: \"kubernetes.io/projected/1acb7079-2d25-4f03-8b17-dfc06e4bebda-kube-api-access-l2jqn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:48.074915 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:48.074805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:48.290719 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:48.290695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"]
Jul 28 01:46:48.291650 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:46:48.291622    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acb7079_2d25_4f03_8b17_dfc06e4bebda.slice/crio-5e82bc12875abdaffa2e3519b46de9e7d15ce24b6b9fb3a8f00bdb7f27b1928a WatchSource:0}: Error finding container 5e82bc12875abdaffa2e3519b46de9e7d15ce24b6b9fb3a8f00bdb7f27b1928a: Status 404 returned error can't find the container with id 5e82bc12875abdaffa2e3519b46de9e7d15ce24b6b9fb3a8f00bdb7f27b1928a
Jul 28 01:46:48.463923 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:48.463809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" event={"ID":"1acb7079-2d25-4f03-8b17-dfc06e4bebda","Type":"ContainerStarted","Data":"5e82bc12875abdaffa2e3519b46de9e7d15ce24b6b9fb3a8f00bdb7f27b1928a"}
Jul 28 01:46:50.480024 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:50.479994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" event={"ID":"1acb7079-2d25-4f03-8b17-dfc06e4bebda","Type":"ContainerStarted","Data":"02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"}
Jul 28 01:46:50.499821 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:50.499770    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" podStartSLOduration=1.775334774 podStartE2EDuration="3.499755034s" totalImagesPullingTime="1.72442026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:50.49694965 +0000 UTC m=+1261.331697198" watchObservedRunningTime="2026-07-28 01:46:50.499755034 +0000 UTC m=+1261.334502581"
Jul 28 01:46:51.075287 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:51.075253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:58.075815 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:58.075776    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:58.088406 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:58.088381    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:46:58.535477 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:46:58.535446    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:47:21.637117 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:21.637082    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"]
Jul 28 01:47:21.646716 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:47:21.646675    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:47:21.646716 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:47:21.646716 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:47:21.646716 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954" cmd=["/bin/sleep","15"]
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:21.646735    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" message="" err=<
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:47:21.646749    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" podUID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" containerName="main" containerID="cri-o://02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"
Jul 28 01:47:21.647059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:21.646795    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" podUID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" containerName="main" containerID="cri-o://02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954" gracePeriod=60
Jul 28 01:47:22.422229 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.422202    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:47:22.500371 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-home\") pod \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") "
Jul 28 01:47:22.500371 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tls-certs\") pod \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") "
Jul 28 01:47:22.500589 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tmp-dir\") pod \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") "
Jul 28 01:47:22.500589 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2jqn\" (UniqueName: \"kubernetes.io/projected/1acb7079-2d25-4f03-8b17-dfc06e4bebda-kube-api-access-l2jqn\") pod \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") "
Jul 28 01:47:22.500589 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-dshm\") pod \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") "
Jul 28 01:47:22.500589 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-model-cache\") pod \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\" (UID: \"1acb7079-2d25-4f03-8b17-dfc06e4bebda\") "
Jul 28 01:47:22.500813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-home" (OuterVolumeSpecName: "home") pod "1acb7079-2d25-4f03-8b17-dfc06e4bebda" (UID: "1acb7079-2d25-4f03-8b17-dfc06e4bebda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:22.500813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1acb7079-2d25-4f03-8b17-dfc06e4bebda" (UID: "1acb7079-2d25-4f03-8b17-dfc06e4bebda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:22.500952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-model-cache" (OuterVolumeSpecName: "model-cache") pod "1acb7079-2d25-4f03-8b17-dfc06e4bebda" (UID: "1acb7079-2d25-4f03-8b17-dfc06e4bebda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:22.500952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500903    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:47:22.500952 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.500923    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:47:22.502745 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.502723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-dshm" (OuterVolumeSpecName: "dshm") pod "1acb7079-2d25-4f03-8b17-dfc06e4bebda" (UID: "1acb7079-2d25-4f03-8b17-dfc06e4bebda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:22.502745 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.502732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1acb7079-2d25-4f03-8b17-dfc06e4bebda" (UID: "1acb7079-2d25-4f03-8b17-dfc06e4bebda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:22.502946 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.502744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acb7079-2d25-4f03-8b17-dfc06e4bebda-kube-api-access-l2jqn" (OuterVolumeSpecName: "kube-api-access-l2jqn") pod "1acb7079-2d25-4f03-8b17-dfc06e4bebda" (UID: "1acb7079-2d25-4f03-8b17-dfc06e4bebda"). InnerVolumeSpecName "kube-api-access-l2jqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:22.602156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.602055    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7079-2d25-4f03-8b17-dfc06e4bebda-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:47:22.602156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.602090    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jqn\" (UniqueName: \"kubernetes.io/projected/1acb7079-2d25-4f03-8b17-dfc06e4bebda-kube-api-access-l2jqn\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:47:22.602156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.602102    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:47:22.602156 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.602113    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1acb7079-2d25-4f03-8b17-dfc06e4bebda-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:47:22.661443 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.661405    2577 generic.go:356] "Generic (PLEG): container finished" podID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" containerID="02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954" exitCode=0
Jul 28 01:47:22.661894 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.661454    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"
Jul 28 01:47:22.661894 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.661484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" event={"ID":"1acb7079-2d25-4f03-8b17-dfc06e4bebda","Type":"ContainerDied","Data":"02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"}
Jul 28 01:47:22.661894 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.661537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v" event={"ID":"1acb7079-2d25-4f03-8b17-dfc06e4bebda","Type":"ContainerDied","Data":"5e82bc12875abdaffa2e3519b46de9e7d15ce24b6b9fb3a8f00bdb7f27b1928a"}
Jul 28 01:47:22.661894 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.661559    2577 scope.go:117] "RemoveContainer" containerID="02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"
Jul 28 01:47:22.672900 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.672881    2577 scope.go:117] "RemoveContainer" containerID="02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"
Jul 28 01:47:22.673206 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:47:22.673184    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954\": container with ID starting with 02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954 not found: ID does not exist" containerID="02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"
Jul 28 01:47:22.673270 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.673215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954"} err="failed to get container status \"02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954\": rpc error: code = NotFound desc = could not find container \"02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954\": container with ID starting with 02a0964b29c8e6651a84b497c0f6e7943c875f1b9d2da0ca6fc109fa36e57954 not found: ID does not exist"
Jul 28 01:47:22.686126 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.686099    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"]
Jul 28 01:47:22.690158 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:22.690134    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fsqm2v"]
Jul 28 01:47:23.697724 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:23.697677    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" path="/var/lib/kubelet/pods/1acb7079-2d25-4f03-8b17-dfc06e4bebda/volumes"
Jul 28 01:47:53.043677 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.043646    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"]
Jul 28 01:47:53.044097 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.044083    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" containerName="main"
Jul 28 01:47:53.044145 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.044099    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" containerName="main"
Jul 28 01:47:53.044181 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.044168    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acb7079-2d25-4f03-8b17-dfc06e4bebda" containerName="main"
Jul 28 01:47:53.048348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.048332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.050897 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.050849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 28 01:47:53.051874 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.051840    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 28 01:47:53.051942 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.051847    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-jqlmr\""
Jul 28 01:47:53.051942 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.051847    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 28 01:47:53.058680 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.058655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"]
Jul 28 01:47:53.174245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.174217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.174245 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.174248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdlg\" (UniqueName: \"kubernetes.io/projected/01007af0-e2c7-4a49-87c9-cde6c8373eae-kube-api-access-bhdlg\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.174438 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.174277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.174438 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.174301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.174438 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.174320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01007af0-e2c7-4a49-87c9-cde6c8373eae-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.174438 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.174410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.275576 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.275759 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.275759 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdlg\" (UniqueName: \"kubernetes.io/projected/01007af0-e2c7-4a49-87c9-cde6c8373eae-kube-api-access-bhdlg\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.275918 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.275918 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.275918 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01007af0-e2c7-4a49-87c9-cde6c8373eae-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.276094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.275966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.276094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.276020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.276168 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.276121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.278200 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.278177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.278369 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.278352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01007af0-e2c7-4a49-87c9-cde6c8373eae-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.286989 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.286961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdlg\" (UniqueName: \"kubernetes.io/projected/01007af0-e2c7-4a49-87c9-cde6c8373eae-kube-api-access-bhdlg\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.290003 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.289981    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"]
Jul 28 01:47:53.294288 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.294236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.298959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.298938    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-ttxlw\""
Jul 28 01:47:53.309210 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.309186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"]
Jul 28 01:47:53.359180 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.359155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:47:53.377478 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.377453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbt8\" (UniqueName: \"kubernetes.io/projected/f4c3e3ab-e1da-419a-b6bd-aed047993977-kube-api-access-ztbt8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.377579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.377501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c3e3ab-e1da-419a-b6bd-aed047993977-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.478672 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.478636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbt8\" (UniqueName: \"kubernetes.io/projected/f4c3e3ab-e1da-419a-b6bd-aed047993977-kube-api-access-ztbt8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.478813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.478689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c3e3ab-e1da-419a-b6bd-aed047993977-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.481396 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.481367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c3e3ab-e1da-419a-b6bd-aed047993977-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.487039 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.487014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbt8\" (UniqueName: \"kubernetes.io/projected/f4c3e3ab-e1da-419a-b6bd-aed047993977-kube-api-access-ztbt8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.503585 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.503564    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"]
Jul 28 01:47:53.504532 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:47:53.504513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01007af0_e2c7_4a49_87c9_cde6c8373eae.slice/crio-5f4bc9c2622a8a7ee46dbdbefab517fc374632b4a291c78e79208d4ccccde24a WatchSource:0}: Error finding container 5f4bc9c2622a8a7ee46dbdbefab517fc374632b4a291c78e79208d4ccccde24a: Status 404 returned error can't find the container with id 5f4bc9c2622a8a7ee46dbdbefab517fc374632b4a291c78e79208d4ccccde24a
Jul 28 01:47:53.612948 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.612918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.750945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.750919    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"]
Jul 28 01:47:53.752037 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:47:53.752004    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c3e3ab_e1da_419a_b6bd_aed047993977.slice/crio-09f4ccb908ef1f586f0c09e6b08a92732a566691832d310219ecd2cf6d12a239 WatchSource:0}: Error finding container 09f4ccb908ef1f586f0c09e6b08a92732a566691832d310219ecd2cf6d12a239: Status 404 returned error can't find the container with id 09f4ccb908ef1f586f0c09e6b08a92732a566691832d310219ecd2cf6d12a239
Jul 28 01:47:53.849307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.849256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" event={"ID":"f4c3e3ab-e1da-419a-b6bd-aed047993977","Type":"ContainerStarted","Data":"e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7"}
Jul 28 01:47:53.849476 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.849337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:47:53.849476 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.849359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" event={"ID":"f4c3e3ab-e1da-419a-b6bd-aed047993977","Type":"ContainerStarted","Data":"09f4ccb908ef1f586f0c09e6b08a92732a566691832d310219ecd2cf6d12a239"}
Jul 28 01:47:53.850856 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.850832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" event={"ID":"01007af0-e2c7-4a49-87c9-cde6c8373eae","Type":"ContainerStarted","Data":"672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"}
Jul 28 01:47:53.851000 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.850883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" event={"ID":"01007af0-e2c7-4a49-87c9-cde6c8373eae","Type":"ContainerStarted","Data":"5f4bc9c2622a8a7ee46dbdbefab517fc374632b4a291c78e79208d4ccccde24a"}
Jul 28 01:47:53.867419 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.866571    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" podStartSLOduration=0.86655287 podStartE2EDuration="866.55287ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:53.866174341 +0000 UTC m=+1324.700921910" watchObservedRunningTime="2026-07-28 01:47:53.86655287 +0000 UTC m=+1324.701300417"
Jul 28 01:47:53.886675 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:53.886615    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" podStartSLOduration=0.886597337 podStartE2EDuration="886.597337ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:53.886041993 +0000 UTC m=+1324.720789540" watchObservedRunningTime="2026-07-28 01:47:53.886597337 +0000 UTC m=+1324.721344884"
Jul 28 01:47:54.359981 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:47:54.359954    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:48:03.359550 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:03.359515    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:48:03.376140 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:03.376113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:48:03.915552 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:03.915520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:48:24.858259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:24.858163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:48:26.315491 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.315457    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"]
Jul 28 01:48:26.321084 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.321050    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"]
Jul 28 01:48:26.331960 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:48:26.331921    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:48:26.331960 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:48:26.331960 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:48:26.331960 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327" cmd=["/bin/sleep","15"]
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.331979    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" message="" err=<
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:48:26.331996    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" podUID="01007af0-e2c7-4a49-87c9-cde6c8373eae" containerName="main" containerID="cri-o://672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"
Jul 28 01:48:26.332257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.332054    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" podUID="01007af0-e2c7-4a49-87c9-cde6c8373eae" containerName="main" containerID="cri-o://672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327" gracePeriod=60
Jul 28 01:48:26.691112 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.691089    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:48:26.785056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785025    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-tmp-dir\") pod \"01007af0-e2c7-4a49-87c9-cde6c8373eae\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") "
Jul 28 01:48:26.785056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-dshm\") pod \"01007af0-e2c7-4a49-87c9-cde6c8373eae\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") "
Jul 28 01:48:26.785252 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-model-cache\") pod \"01007af0-e2c7-4a49-87c9-cde6c8373eae\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") "
Jul 28 01:48:26.785252 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhdlg\" (UniqueName: \"kubernetes.io/projected/01007af0-e2c7-4a49-87c9-cde6c8373eae-kube-api-access-bhdlg\") pod \"01007af0-e2c7-4a49-87c9-cde6c8373eae\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") "
Jul 28 01:48:26.785252 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-home\") pod \"01007af0-e2c7-4a49-87c9-cde6c8373eae\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") "
Jul 28 01:48:26.785252 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785245    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01007af0-e2c7-4a49-87c9-cde6c8373eae-tls-certs\") pod \"01007af0-e2c7-4a49-87c9-cde6c8373eae\" (UID: \"01007af0-e2c7-4a49-87c9-cde6c8373eae\") "
Jul 28 01:48:26.785462 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01007af0-e2c7-4a49-87c9-cde6c8373eae" (UID: "01007af0-e2c7-4a49-87c9-cde6c8373eae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:48:26.785522 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-model-cache" (OuterVolumeSpecName: "model-cache") pod "01007af0-e2c7-4a49-87c9-cde6c8373eae" (UID: "01007af0-e2c7-4a49-87c9-cde6c8373eae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:48:26.785580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-home" (OuterVolumeSpecName: "home") pod "01007af0-e2c7-4a49-87c9-cde6c8373eae" (UID: "01007af0-e2c7-4a49-87c9-cde6c8373eae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:48:26.785640 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785619    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:26.785701 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785643    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:26.785701 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.785653    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:26.787374 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.787348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-dshm" (OuterVolumeSpecName: "dshm") pod "01007af0-e2c7-4a49-87c9-cde6c8373eae" (UID: "01007af0-e2c7-4a49-87c9-cde6c8373eae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:48:26.787500 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.787396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01007af0-e2c7-4a49-87c9-cde6c8373eae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01007af0-e2c7-4a49-87c9-cde6c8373eae" (UID: "01007af0-e2c7-4a49-87c9-cde6c8373eae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:26.787500 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.787451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01007af0-e2c7-4a49-87c9-cde6c8373eae-kube-api-access-bhdlg" (OuterVolumeSpecName: "kube-api-access-bhdlg") pod "01007af0-e2c7-4a49-87c9-cde6c8373eae" (UID: "01007af0-e2c7-4a49-87c9-cde6c8373eae"). InnerVolumeSpecName "kube-api-access-bhdlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:26.886912 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.886805    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdlg\" (UniqueName: \"kubernetes.io/projected/01007af0-e2c7-4a49-87c9-cde6c8373eae-kube-api-access-bhdlg\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:26.886912 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.886833    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01007af0-e2c7-4a49-87c9-cde6c8373eae-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:26.886912 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:26.886843    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01007af0-e2c7-4a49-87c9-cde6c8373eae-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:27.031428 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.031394    2577 generic.go:356] "Generic (PLEG): container finished" podID="01007af0-e2c7-4a49-87c9-cde6c8373eae" containerID="672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327" exitCode=0
Jul 28 01:48:27.031625 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.031437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"
Jul 28 01:48:27.031625 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.031462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" event={"ID":"01007af0-e2c7-4a49-87c9-cde6c8373eae","Type":"ContainerDied","Data":"672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"}
Jul 28 01:48:27.031625 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.031513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp" event={"ID":"01007af0-e2c7-4a49-87c9-cde6c8373eae","Type":"ContainerDied","Data":"5f4bc9c2622a8a7ee46dbdbefab517fc374632b4a291c78e79208d4ccccde24a"}
Jul 28 01:48:27.031625 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.031540    2577 scope.go:117] "RemoveContainer" containerID="672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"
Jul 28 01:48:27.041192 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.041176    2577 scope.go:117] "RemoveContainer" containerID="672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"
Jul 28 01:48:27.041448 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:48:27.041430    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327\": container with ID starting with 672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327 not found: ID does not exist" containerID="672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"
Jul 28 01:48:27.041491 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.041458    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327"} err="failed to get container status \"672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327\": rpc error: code = NotFound desc = could not find container \"672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327\": container with ID starting with 672e88c6029ab87f248fdd82bf13e5b2b32b7dcdf772ab96c2f12c2f19fc3327 not found: ID does not exist"
Jul 28 01:48:27.055276 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.055232    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"]
Jul 28 01:48:27.056854 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.056835    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kclzp"]
Jul 28 01:48:27.697948 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:27.697915    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01007af0-e2c7-4a49-87c9-cde6c8373eae" path="/var/lib/kubelet/pods/01007af0-e2c7-4a49-87c9-cde6c8373eae/volumes"
Jul 28 01:48:34.858287 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:34.858247    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" podUID="f4c3e3ab-e1da-419a-b6bd-aed047993977" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:48:41.340345 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.340306    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" podUID="f4c3e3ab-e1da-419a-b6bd-aed047993977" containerName="main" containerID="cri-o://e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7" gracePeriod=45
Jul 28 01:48:41.600304 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.600247    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:48:41.715742 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.715708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztbt8\" (UniqueName: \"kubernetes.io/projected/f4c3e3ab-e1da-419a-b6bd-aed047993977-kube-api-access-ztbt8\") pod \"f4c3e3ab-e1da-419a-b6bd-aed047993977\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") "
Jul 28 01:48:41.715742 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.715744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c3e3ab-e1da-419a-b6bd-aed047993977-tls-certs\") pod \"f4c3e3ab-e1da-419a-b6bd-aed047993977\" (UID: \"f4c3e3ab-e1da-419a-b6bd-aed047993977\") "
Jul 28 01:48:41.717825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.717794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c3e3ab-e1da-419a-b6bd-aed047993977-kube-api-access-ztbt8" (OuterVolumeSpecName: "kube-api-access-ztbt8") pod "f4c3e3ab-e1da-419a-b6bd-aed047993977" (UID: "f4c3e3ab-e1da-419a-b6bd-aed047993977"). InnerVolumeSpecName "kube-api-access-ztbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:41.717825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.717808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c3e3ab-e1da-419a-b6bd-aed047993977-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4c3e3ab-e1da-419a-b6bd-aed047993977" (UID: "f4c3e3ab-e1da-419a-b6bd-aed047993977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:41.817193 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.817152    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbt8\" (UniqueName: \"kubernetes.io/projected/f4c3e3ab-e1da-419a-b6bd-aed047993977-kube-api-access-ztbt8\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:41.817193 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:41.817186    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c3e3ab-e1da-419a-b6bd-aed047993977-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:48:42.115775 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.115737    2577 generic.go:356] "Generic (PLEG): container finished" podID="f4c3e3ab-e1da-419a-b6bd-aed047993977" containerID="e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7" exitCode=0
Jul 28 01:48:42.115971 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.115780    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"
Jul 28 01:48:42.115971 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.115829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" event={"ID":"f4c3e3ab-e1da-419a-b6bd-aed047993977","Type":"ContainerDied","Data":"e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7"}
Jul 28 01:48:42.115971 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.115903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg" event={"ID":"f4c3e3ab-e1da-419a-b6bd-aed047993977","Type":"ContainerDied","Data":"09f4ccb908ef1f586f0c09e6b08a92732a566691832d310219ecd2cf6d12a239"}
Jul 28 01:48:42.115971 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.115919    2577 scope.go:117] "RemoveContainer" containerID="e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7"
Jul 28 01:48:42.125193 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.125175    2577 scope.go:117] "RemoveContainer" containerID="e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7"
Jul 28 01:48:42.125444 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:48:42.125427    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7\": container with ID starting with e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7 not found: ID does not exist" containerID="e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7"
Jul 28 01:48:42.125500 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.125452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7"} err="failed to get container status \"e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7\": rpc error: code = NotFound desc = could not find container \"e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7\": container with ID starting with e4b42152ec976cc9e9d980766d192db49d82fbf87fc1cf39542eb7cf310c53a7 not found: ID does not exist"
Jul 28 01:48:42.136243 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.136219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"]
Jul 28 01:48:42.138786 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:42.138762    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587bdpwg"]
Jul 28 01:48:43.697829 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:43.697795    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c3e3ab-e1da-419a-b6bd-aed047993977" path="/var/lib/kubelet/pods/f4c3e3ab-e1da-419a-b6bd-aed047993977/volumes"
Jul 28 01:48:53.927564 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.927531    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"]
Jul 28 01:48:53.927959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.927911    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c3e3ab-e1da-419a-b6bd-aed047993977" containerName="main"
Jul 28 01:48:53.927959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.927923    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c3e3ab-e1da-419a-b6bd-aed047993977" containerName="main"
Jul 28 01:48:53.927959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.927934    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01007af0-e2c7-4a49-87c9-cde6c8373eae" containerName="main"
Jul 28 01:48:53.927959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.927940    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01007af0-e2c7-4a49-87c9-cde6c8373eae" containerName="main"
Jul 28 01:48:53.928097 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.928006    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01007af0-e2c7-4a49-87c9-cde6c8373eae" containerName="main"
Jul 28 01:48:53.928097 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.928012    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c3e3ab-e1da-419a-b6bd-aed047993977" containerName="main"
Jul 28 01:48:53.932430 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.932414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:53.936055 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.936026    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 01:48:53.936179 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.936058    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 01:48:53.936179 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.936069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 01:48:53.936179 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.936140    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-vvkxm\""
Jul 28 01:48:53.940402 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:53.940378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"]
Jul 28 01:48:54.023047 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.023011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4697\" (UniqueName: \"kubernetes.io/projected/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-kube-api-access-w4697\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.023220 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.023063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.023220 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.023087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.023220 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.023121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.023220 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.023139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.023372 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.023265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124153 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124319 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4697\" (UniqueName: \"kubernetes.io/projected/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-kube-api-access-w4697\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124319 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124401 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124401 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124401 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.124700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.124696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.126572 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.126551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.126789 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.126772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.133275 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.133257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4697\" (UniqueName: \"kubernetes.io/projected/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-kube-api-access-w4697\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.204083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.204011    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"]
Jul 28 01:48:54.207715 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.207697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.211487 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.211467    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-tk5n9\""
Jul 28 01:48:54.219499 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.219474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"]
Jul 28 01:48:54.244507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.244483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:48:54.326557 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.326521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w52c\" (UniqueName: \"kubernetes.io/projected/9eb77ae6-cda5-410a-bf72-af106c0164da-kube-api-access-9w52c\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.326700 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.326670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb77ae6-cda5-410a-bf72-af106c0164da-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.382002 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.381971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"]
Jul 28 01:48:54.385375 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:48:54.385343    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13edcb28_d465_4acb_ae07_bf9c4fcea8b2.slice/crio-ad118ba75d27278eca6bf0bdc9cd6f1006a9085704a7b9fbbe9baff94798fb90 WatchSource:0}: Error finding container ad118ba75d27278eca6bf0bdc9cd6f1006a9085704a7b9fbbe9baff94798fb90: Status 404 returned error can't find the container with id ad118ba75d27278eca6bf0bdc9cd6f1006a9085704a7b9fbbe9baff94798fb90
Jul 28 01:48:54.427887 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.427840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w52c\" (UniqueName: \"kubernetes.io/projected/9eb77ae6-cda5-410a-bf72-af106c0164da-kube-api-access-9w52c\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.428026 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.427979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb77ae6-cda5-410a-bf72-af106c0164da-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.430277 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.430251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb77ae6-cda5-410a-bf72-af106c0164da-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.436271 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.436247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w52c\" (UniqueName: \"kubernetes.io/projected/9eb77ae6-cda5-410a-bf72-af106c0164da-kube-api-access-9w52c\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.519136 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.519101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:54.660174 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:54.660148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"]
Jul 28 01:48:54.661813 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:48:54.661781    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb77ae6_cda5_410a_bf72_af106c0164da.slice/crio-3cb1da429bd9856d8fcbee556fe75a2e331b8716dde18ed47d68080d0dfbb99d WatchSource:0}: Error finding container 3cb1da429bd9856d8fcbee556fe75a2e331b8716dde18ed47d68080d0dfbb99d: Status 404 returned error can't find the container with id 3cb1da429bd9856d8fcbee556fe75a2e331b8716dde18ed47d68080d0dfbb99d
Jul 28 01:48:55.194174 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.194131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" event={"ID":"9eb77ae6-cda5-410a-bf72-af106c0164da","Type":"ContainerStarted","Data":"2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"}
Jul 28 01:48:55.194730 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.194241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" event={"ID":"9eb77ae6-cda5-410a-bf72-af106c0164da","Type":"ContainerStarted","Data":"3cb1da429bd9856d8fcbee556fe75a2e331b8716dde18ed47d68080d0dfbb99d"}
Jul 28 01:48:55.194730 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.194672    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:48:55.196616 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.196550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" event={"ID":"13edcb28-d465-4acb-ae07-bf9c4fcea8b2","Type":"ContainerStarted","Data":"ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"}
Jul 28 01:48:55.196616 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.196599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" event={"ID":"13edcb28-d465-4acb-ae07-bf9c4fcea8b2","Type":"ContainerStarted","Data":"ad118ba75d27278eca6bf0bdc9cd6f1006a9085704a7b9fbbe9baff94798fb90"}
Jul 28 01:48:55.215463 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.215414    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" podStartSLOduration=1.2153994830000001 podStartE2EDuration="1.215399483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:55.213756741 +0000 UTC m=+1386.048504290" watchObservedRunningTime="2026-07-28 01:48:55.215399483 +0000 UTC m=+1386.050147029"
Jul 28 01:48:55.238525 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.238476    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" podStartSLOduration=2.238461093 podStartE2EDuration="2.238461093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:55.233679436 +0000 UTC m=+1386.068426981" watchObservedRunningTime="2026-07-28 01:48:55.238461093 +0000 UTC m=+1386.073208640"
Jul 28 01:48:55.245518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:48:55.245494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:49:04.245042 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:04.245009    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:49:04.257502 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:04.257478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:49:05.257185 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:05.257153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:49:26.204360 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:26.204331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:49:27.580724 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:27.580689    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"]
Jul 28 01:49:27.592649 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:27.592619    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"]
Jul 28 01:49:27.596008 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:49:27.595969    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:49:27.596008 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:49:27.596008 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:49:27.596008 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af" cmd=["/bin/sleep","15"]
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:27.596031    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" message="" err=<
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:49:27.596049    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" podUID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" containerName="main" containerID="cri-o://ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"
Jul 28 01:49:27.596303 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:27.596093    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" podUID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" containerName="main" containerID="cri-o://ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af" gracePeriod=60
Jul 28 01:49:27.957024 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:27.957004    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:49:28.027905 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.027851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-home\") pod \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") "
Jul 28 01:49:28.028083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.027911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tls-certs\") pod \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") "
Jul 28 01:49:28.028083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.027937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tmp-dir\") pod \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") "
Jul 28 01:49:28.028083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.027981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-dshm\") pod \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") "
Jul 28 01:49:28.028083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-model-cache\") pod \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") "
Jul 28 01:49:28.028083 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4697\" (UniqueName: \"kubernetes.io/projected/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-kube-api-access-w4697\") pod \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\" (UID: \"13edcb28-d465-4acb-ae07-bf9c4fcea8b2\") "
Jul 28 01:49:28.028367 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-home" (OuterVolumeSpecName: "home") pod "13edcb28-d465-4acb-ae07-bf9c4fcea8b2" (UID: "13edcb28-d465-4acb-ae07-bf9c4fcea8b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:28.028367 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13edcb28-d465-4acb-ae07-bf9c4fcea8b2" (UID: "13edcb28-d465-4acb-ae07-bf9c4fcea8b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:28.028367 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "13edcb28-d465-4acb-ae07-bf9c4fcea8b2" (UID: "13edcb28-d465-4acb-ae07-bf9c4fcea8b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:28.028520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028418    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:28.028520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028434    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:28.028520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.028447    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:28.030293 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.030265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-dshm" (OuterVolumeSpecName: "dshm") pod "13edcb28-d465-4acb-ae07-bf9c4fcea8b2" (UID: "13edcb28-d465-4acb-ae07-bf9c4fcea8b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:28.030293 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.030285    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-kube-api-access-w4697" (OuterVolumeSpecName: "kube-api-access-w4697") pod "13edcb28-d465-4acb-ae07-bf9c4fcea8b2" (UID: "13edcb28-d465-4acb-ae07-bf9c4fcea8b2"). InnerVolumeSpecName "kube-api-access-w4697". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:49:28.030468 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.030307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13edcb28-d465-4acb-ae07-bf9c4fcea8b2" (UID: "13edcb28-d465-4acb-ae07-bf9c4fcea8b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:28.129311 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.129234    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4697\" (UniqueName: \"kubernetes.io/projected/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-kube-api-access-w4697\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:28.129311 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.129261    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:28.129311 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.129273    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13edcb28-d465-4acb-ae07-bf9c4fcea8b2-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:28.392454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.392368    2577 generic.go:356] "Generic (PLEG): container finished" podID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" containerID="ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af" exitCode=0
Jul 28 01:49:28.392454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.392430    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"
Jul 28 01:49:28.392454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.392444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" event={"ID":"13edcb28-d465-4acb-ae07-bf9c4fcea8b2","Type":"ContainerDied","Data":"ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"}
Jul 28 01:49:28.392668 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.392519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm" event={"ID":"13edcb28-d465-4acb-ae07-bf9c4fcea8b2","Type":"ContainerDied","Data":"ad118ba75d27278eca6bf0bdc9cd6f1006a9085704a7b9fbbe9baff94798fb90"}
Jul 28 01:49:28.392668 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.392540    2577 scope.go:117] "RemoveContainer" containerID="ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"
Jul 28 01:49:28.402312 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.402288    2577 scope.go:117] "RemoveContainer" containerID="ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"
Jul 28 01:49:28.402616 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:49:28.402597    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af\": container with ID starting with ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af not found: ID does not exist" containerID="ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"
Jul 28 01:49:28.402686 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.402624    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af"} err="failed to get container status \"ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af\": rpc error: code = NotFound desc = could not find container \"ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af\": container with ID starting with ad028768c6be50a7d01cc4e5e62f94a79b15e6dd4462dd11e939ff63f07771af not found: ID does not exist"
Jul 28 01:49:28.414682 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.414651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"]
Jul 28 01:49:28.419564 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:28.419543    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-7rcdm"]
Jul 28 01:49:29.699062 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:29.699026    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" path="/var/lib/kubelet/pods/13edcb28-d465-4acb-ae07-bf9c4fcea8b2/volumes"
Jul 28 01:49:36.204313 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:36.204275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:49:38.950313 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:38.950271    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 28 01:49:38.950633 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:49:38.950316    2577 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerName="main" containerID="cri-o://2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"
Jul 28 01:49:38.950633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:38.950366    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerName="main" containerID="cri-o://2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846" gracePeriod=49
Jul 28 01:49:39.101515 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.101494    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n_9eb77ae6-cda5-410a-bf72-af106c0164da/main/0.log"
Jul 28 01:49:39.103734 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.103720    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:49:39.232287 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.232260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb77ae6-cda5-410a-bf72-af106c0164da-tls-certs\") pod \"9eb77ae6-cda5-410a-bf72-af106c0164da\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") "
Jul 28 01:49:39.232428 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.232294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w52c\" (UniqueName: \"kubernetes.io/projected/9eb77ae6-cda5-410a-bf72-af106c0164da-kube-api-access-9w52c\") pod \"9eb77ae6-cda5-410a-bf72-af106c0164da\" (UID: \"9eb77ae6-cda5-410a-bf72-af106c0164da\") "
Jul 28 01:49:39.234336 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.234302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb77ae6-cda5-410a-bf72-af106c0164da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9eb77ae6-cda5-410a-bf72-af106c0164da" (UID: "9eb77ae6-cda5-410a-bf72-af106c0164da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:39.234443 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.234310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb77ae6-cda5-410a-bf72-af106c0164da-kube-api-access-9w52c" (OuterVolumeSpecName: "kube-api-access-9w52c") pod "9eb77ae6-cda5-410a-bf72-af106c0164da" (UID: "9eb77ae6-cda5-410a-bf72-af106c0164da"). InnerVolumeSpecName "kube-api-access-9w52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:49:39.333830 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.333798    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb77ae6-cda5-410a-bf72-af106c0164da-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:39.333830 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.333823    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w52c\" (UniqueName: \"kubernetes.io/projected/9eb77ae6-cda5-410a-bf72-af106c0164da-kube-api-access-9w52c\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:49:39.455388 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.455362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n_9eb77ae6-cda5-410a-bf72-af106c0164da/main/0.log"
Jul 28 01:49:39.455546 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.455398    2577 generic.go:356] "Generic (PLEG): container finished" podID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerID="2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846" exitCode=1
Jul 28 01:49:39.455546 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.455452    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"
Jul 28 01:49:39.455546 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.455493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" event={"ID":"9eb77ae6-cda5-410a-bf72-af106c0164da","Type":"ContainerDied","Data":"2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"}
Jul 28 01:49:39.455657 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.455568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n" event={"ID":"9eb77ae6-cda5-410a-bf72-af106c0164da","Type":"ContainerDied","Data":"3cb1da429bd9856d8fcbee556fe75a2e331b8716dde18ed47d68080d0dfbb99d"}
Jul 28 01:49:39.455657 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.455591    2577 scope.go:117] "RemoveContainer" containerID="2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"
Jul 28 01:49:39.464795 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.464778    2577 scope.go:117] "RemoveContainer" containerID="2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"
Jul 28 01:49:39.465084 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:49:39.465065    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846\": container with ID starting with 2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846 not found: ID does not exist" containerID="2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"
Jul 28 01:49:39.465169 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.465092    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846"} err="failed to get container status \"2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846\": rpc error: code = NotFound desc = could not find container \"2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846\": container with ID starting with 2c46ac96985315e88a91befc922186c3514fde4da0c70ce293ed08d6d5f83846 not found: ID does not exist"
Jul 28 01:49:39.478511 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.478483    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"]
Jul 28 01:49:39.481870 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.481836    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkzh8n"]
Jul 28 01:49:39.699581 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:49:39.699506    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" path="/var/lib/kubelet/pods/9eb77ae6-cda5-410a-bf72-af106c0164da/volumes"
Jul 28 01:50:51.535149 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:50:51.535049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:50:51.538788 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:50:51.538431    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:51:09.346899 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.346848    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"]
Jul 28 01:51:09.347422 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.347407    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" containerName="main"
Jul 28 01:51:09.347470 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.347425    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" containerName="main"
Jul 28 01:51:09.347470 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.347446    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerName="main"
Jul 28 01:51:09.347470 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.347454    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerName="main"
Jul 28 01:51:09.347570 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.347559    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="13edcb28-d465-4acb-ae07-bf9c4fcea8b2" containerName="main"
Jul 28 01:51:09.347609 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.347577    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb77ae6-cda5-410a-bf72-af106c0164da" containerName="main"
Jul 28 01:51:09.351215 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.351194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.355098 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.355075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pvjf2\""
Jul 28 01:51:09.355098 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.355094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 01:51:09.355286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.355097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 01:51:09.355286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.355128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 01:51:09.355286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.355176    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-88fgq\""
Jul 28 01:51:09.361171 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.361151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"]
Jul 28 01:51:09.411332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.411302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxls8\" (UniqueName: \"kubernetes.io/projected/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-kube-api-access-pxls8\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.411456 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.411337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.411456 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.411366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.411456 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.411423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.411588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.411497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.411588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.411542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxls8\" (UniqueName: \"kubernetes.io/projected/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-kube-api-access-pxls8\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512520 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512730 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512856 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.512972 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.512952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.513016 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.513006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.514615 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.514590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.514918 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.514900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.521407 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.521384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxls8\" (UniqueName: \"kubernetes.io/projected/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-kube-api-access-pxls8\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.661975 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.661900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:09.828289 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.828262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"]
Jul 28 01:51:09.829543 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:51:09.829521    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd2dcad_f8ae_496d_bf4d_2d0feac8b9f4.slice/crio-a1cbd55a907667c3e9bf12dbf660eb14df94dc64bc4ca72263e6e53943d65ef6 WatchSource:0}: Error finding container a1cbd55a907667c3e9bf12dbf660eb14df94dc64bc4ca72263e6e53943d65ef6: Status 404 returned error can't find the container with id a1cbd55a907667c3e9bf12dbf660eb14df94dc64bc4ca72263e6e53943d65ef6
Jul 28 01:51:09.831577 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.831561    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:51:09.939624 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:09.939553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" event={"ID":"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4","Type":"ContainerStarted","Data":"a1cbd55a907667c3e9bf12dbf660eb14df94dc64bc4ca72263e6e53943d65ef6"}
Jul 28 01:51:11.951422 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:11.951388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" event={"ID":"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4","Type":"ContainerStarted","Data":"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"}
Jul 28 01:51:11.951820 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:11.951520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:12.958845 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:12.958809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" event={"ID":"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4","Type":"ContainerStarted","Data":"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"}
Jul 28 01:51:12.981562 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:12.981512    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" podStartSLOduration=2.513878344 podStartE2EDuration="3.981495889s" totalImagesPullingTime="1.467617545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:12.980077197 +0000 UTC m=+1523.814824744" watchObservedRunningTime="2026-07-28 01:51:12.981495889 +0000 UTC m=+1523.816243734"
Jul 28 01:51:13.662727 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:13.662692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:19.663074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:19.663034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:19.675431 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:19.675409    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:20.017088 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:20.017056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:21.024388 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:21.024356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:43.970822 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:43.970784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"]
Jul 28 01:51:43.980517 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:51:43.980480    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:51:43.980517 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:51:43.980517 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:51:43.980517 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6" cmd=["/bin/sleep","15"]
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:43.980527    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" message="" err=<
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:51:43.980540    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="main" containerID="cri-o://e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"
Jul 28 01:51:43.980753 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:43.980573    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="main" containerID="cri-o://e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6" gracePeriod=60
Jul 28 01:51:44.006454 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.006420    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 28 01:51:44.197185 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.197127    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="llm-d-routing-sidecar" containerID="cri-o://dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4" gracePeriod=60
Jul 28 01:51:44.454167 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.454146    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:44.529546 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-dshm\") pod \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") "
Jul 28 01:51:44.529546 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxls8\" (UniqueName: \"kubernetes.io/projected/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-kube-api-access-pxls8\") pod \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") "
Jul 28 01:51:44.529767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-model-cache\") pod \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") "
Jul 28 01:51:44.529767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-home\") pod \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") "
Jul 28 01:51:44.529767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tls-certs\") pod \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") "
Jul 28 01:51:44.529767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tmp-dir\") pod \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\" (UID: \"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4\") "
Jul 28 01:51:44.530031 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" (UID: "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.530031 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.529932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-home" (OuterVolumeSpecName: "home") pod "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" (UID: "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.530125 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.530041    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" (UID: "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.531780 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.531755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-kube-api-access-pxls8" (OuterVolumeSpecName: "kube-api-access-pxls8") pod "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" (UID: "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4"). InnerVolumeSpecName "kube-api-access-pxls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:51:44.532157 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.532136    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" (UID: "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:51:44.532223 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.532139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-dshm" (OuterVolumeSpecName: "dshm") pod "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" (UID: "6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.630588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.630504    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.630588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.630532    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.630588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.630543    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.630588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.630556    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxls8\" (UniqueName: \"kubernetes.io/projected/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-kube-api-access-pxls8\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.630588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.630569    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.630588 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:44.630580    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:51:45.150563 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150524    2577 generic.go:356] "Generic (PLEG): container finished" podID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerID="e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6" exitCode=0
Jul 28 01:51:45.150563 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150552    2577 generic.go:356] "Generic (PLEG): container finished" podID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerID="dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4" exitCode=0
Jul 28 01:51:45.151094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150573    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"
Jul 28 01:51:45.151094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" event={"ID":"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4","Type":"ContainerDied","Data":"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"}
Jul 28 01:51:45.151094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" event={"ID":"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4","Type":"ContainerDied","Data":"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"}
Jul 28 01:51:45.151094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr" event={"ID":"6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4","Type":"ContainerDied","Data":"a1cbd55a907667c3e9bf12dbf660eb14df94dc64bc4ca72263e6e53943d65ef6"}
Jul 28 01:51:45.151094 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.150697    2577 scope.go:117] "RemoveContainer" containerID="e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"
Jul 28 01:51:45.164412 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.164396    2577 scope.go:117] "RemoveContainer" containerID="dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"
Jul 28 01:51:45.172146 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172127    2577 scope.go:117] "RemoveContainer" containerID="e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"
Jul 28 01:51:45.172386 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:51:45.172369    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6\": container with ID starting with e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6 not found: ID does not exist" containerID="e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"
Jul 28 01:51:45.172432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172395    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"} err="failed to get container status \"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6\": rpc error: code = NotFound desc = could not find container \"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6\": container with ID starting with e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6 not found: ID does not exist"
Jul 28 01:51:45.172432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172411    2577 scope.go:117] "RemoveContainer" containerID="dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"
Jul 28 01:51:45.172653 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:51:45.172633    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4\": container with ID starting with dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4 not found: ID does not exist" containerID="dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"
Jul 28 01:51:45.172697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172660    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"} err="failed to get container status \"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4\": rpc error: code = NotFound desc = could not find container \"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4\": container with ID starting with dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4 not found: ID does not exist"
Jul 28 01:51:45.172697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172675    2577 scope.go:117] "RemoveContainer" containerID="e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"
Jul 28 01:51:45.172919 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172901    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6"} err="failed to get container status \"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6\": rpc error: code = NotFound desc = could not find container \"e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6\": container with ID starting with e2a4323be65e8c0d5f8d972455b7864994c4725a8cc61f4ec0ec4ee02dbb5ce6 not found: ID does not exist"
Jul 28 01:51:45.172919 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.172918    2577 scope.go:117] "RemoveContainer" containerID="dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"
Jul 28 01:51:45.173127 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.173110    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4"} err="failed to get container status \"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4\": rpc error: code = NotFound desc = could not find container \"dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4\": container with ID starting with dac09e38010bffa797009b21ea62ed38672f1019715c59f3c1fcdcb5854384d4 not found: ID does not exist"
Jul 28 01:51:45.177225 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.177203    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"]
Jul 28 01:51:45.180810 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.180787    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-tttgr"]
Jul 28 01:51:45.698213 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:51:45.698178    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" path="/var/lib/kubelet/pods/6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4/volumes"
Jul 28 01:52:11.270505 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.270475    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"]
Jul 28 01:52:11.270896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.270876    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="main"
Jul 28 01:52:11.270896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.270888    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="main"
Jul 28 01:52:11.271022 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.270907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="llm-d-routing-sidecar"
Jul 28 01:52:11.271022 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.270912    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="llm-d-routing-sidecar"
Jul 28 01:52:11.271022 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.270995    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="llm-d-routing-sidecar"
Jul 28 01:52:11.271022 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.271014    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd2dcad-f8ae-496d-bf4d-2d0feac8b9f4" containerName="main"
Jul 28 01:52:11.274384 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.274364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.279414 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.279391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-llc24\""
Jul 28 01:52:11.279513 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.279455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 01:52:11.279564 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.279391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 01:52:11.279608 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.279399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 01:52:11.286233 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.286193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"]
Jul 28 01:52:11.361849 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.361816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.362028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.361854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.362028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.361934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.362028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.361988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.362143 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.362033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhkt\" (UniqueName: \"kubernetes.io/projected/d4368c81-6749-45d2-b9d8-2912a0eaf827-kube-api-access-cqhkt\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.362143 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.362112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4368c81-6749-45d2-b9d8-2912a0eaf827-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.462945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.462907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463096 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhkt\" (UniqueName: \"kubernetes.io/projected/d4368c81-6749-45d2-b9d8-2912a0eaf827-kube-api-access-cqhkt\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463146 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4368c81-6749-45d2-b9d8-2912a0eaf827-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463219 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463339 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463339 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463339 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463560 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.463773 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.463743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.465602 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.465580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.465688 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.465616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4368c81-6749-45d2-b9d8-2912a0eaf827-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.476225 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.476198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhkt\" (UniqueName: \"kubernetes.io/projected/d4368c81-6749-45d2-b9d8-2912a0eaf827-kube-api-access-cqhkt\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.586073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.585986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:11.743359 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:11.743328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"]
Jul 28 01:52:11.745198 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:52:11.745173    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4368c81_6749_45d2_b9d8_2912a0eaf827.slice/crio-2d883ce29b1f8c01d6e1533d9195ad84238e0445ccd285b773388aaa18492edf WatchSource:0}: Error finding container 2d883ce29b1f8c01d6e1533d9195ad84238e0445ccd285b773388aaa18492edf: Status 404 returned error can't find the container with id 2d883ce29b1f8c01d6e1533d9195ad84238e0445ccd285b773388aaa18492edf
Jul 28 01:52:12.308706 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:12.308673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" event={"ID":"d4368c81-6749-45d2-b9d8-2912a0eaf827","Type":"ContainerStarted","Data":"e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942"}
Jul 28 01:52:12.309103 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:12.308742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" event={"ID":"d4368c81-6749-45d2-b9d8-2912a0eaf827","Type":"ContainerStarted","Data":"2d883ce29b1f8c01d6e1533d9195ad84238e0445ccd285b773388aaa18492edf"}
Jul 28 01:52:12.330681 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:12.330601    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" podStartSLOduration=1.330582693 podStartE2EDuration="1.330582693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:12.329912266 +0000 UTC m=+1583.164659814" watchObservedRunningTime="2026-07-28 01:52:12.330582693 +0000 UTC m=+1583.165330240"
Jul 28 01:52:12.587155 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:12.587069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:21.586577 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:21.586534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:21.599466 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:21.599437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:22.375285 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:22.375256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:45.261661 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.261573    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"]
Jul 28 01:52:45.273843 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:52:45.273799    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:52:45.273843 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:52:45.273843 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:52:45.273843 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942" cmd=["/bin/sleep","15"]
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.273894    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" message="" err=<
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:52:45.273913    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerName="main" containerID="cri-o://e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942"
Jul 28 01:52:45.274209 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.273960    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerName="main" containerID="cri-o://e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942" gracePeriod=60
Jul 28 01:52:45.365882 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.364168    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 01:52:45.508907 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.508845    2577 generic.go:356] "Generic (PLEG): container finished" podID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerID="e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942" exitCode=0
Jul 28 01:52:45.509057 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.508923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" event={"ID":"d4368c81-6749-45d2-b9d8-2912a0eaf827","Type":"ContainerDied","Data":"e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942"}
Jul 28 01:52:45.644555 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.644526    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:45.789630 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-model-cache\") pod \"d4368c81-6749-45d2-b9d8-2912a0eaf827\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") "
Jul 28 01:52:45.789630 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-tmp-dir\") pod \"d4368c81-6749-45d2-b9d8-2912a0eaf827\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") "
Jul 28 01:52:45.789850 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4368c81-6749-45d2-b9d8-2912a0eaf827-tls-certs\") pod \"d4368c81-6749-45d2-b9d8-2912a0eaf827\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") "
Jul 28 01:52:45.789850 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789693    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-home\") pod \"d4368c81-6749-45d2-b9d8-2912a0eaf827\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") "
Jul 28 01:52:45.789850 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqhkt\" (UniqueName: \"kubernetes.io/projected/d4368c81-6749-45d2-b9d8-2912a0eaf827-kube-api-access-cqhkt\") pod \"d4368c81-6749-45d2-b9d8-2912a0eaf827\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") "
Jul 28 01:52:45.789850 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789805    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-dshm\") pod \"d4368c81-6749-45d2-b9d8-2912a0eaf827\" (UID: \"d4368c81-6749-45d2-b9d8-2912a0eaf827\") "
Jul 28 01:52:45.790099 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789930    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4368c81-6749-45d2-b9d8-2912a0eaf827" (UID: "d4368c81-6749-45d2-b9d8-2912a0eaf827"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.790099 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.789942    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4368c81-6749-45d2-b9d8-2912a0eaf827" (UID: "d4368c81-6749-45d2-b9d8-2912a0eaf827"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.790099 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.790016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-home" (OuterVolumeSpecName: "home") pod "d4368c81-6749-45d2-b9d8-2912a0eaf827" (UID: "d4368c81-6749-45d2-b9d8-2912a0eaf827"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.790262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.790151    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.790262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.790164    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.790262 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.790187    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.791990 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.791958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4368c81-6749-45d2-b9d8-2912a0eaf827-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4368c81-6749-45d2-b9d8-2912a0eaf827" (UID: "d4368c81-6749-45d2-b9d8-2912a0eaf827"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:52:45.792095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.792005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-dshm" (OuterVolumeSpecName: "dshm") pod "d4368c81-6749-45d2-b9d8-2912a0eaf827" (UID: "d4368c81-6749-45d2-b9d8-2912a0eaf827"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.792095 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.792006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4368c81-6749-45d2-b9d8-2912a0eaf827-kube-api-access-cqhkt" (OuterVolumeSpecName: "kube-api-access-cqhkt") pod "d4368c81-6749-45d2-b9d8-2912a0eaf827" (UID: "d4368c81-6749-45d2-b9d8-2912a0eaf827"). InnerVolumeSpecName "kube-api-access-cqhkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:52:45.891315 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.891234    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqhkt\" (UniqueName: \"kubernetes.io/projected/d4368c81-6749-45d2-b9d8-2912a0eaf827-kube-api-access-cqhkt\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.891315 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.891266    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4368c81-6749-45d2-b9d8-2912a0eaf827-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.891315 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:45.891276    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4368c81-6749-45d2-b9d8-2912a0eaf827-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:52:46.516208 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:46.516172    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"
Jul 28 01:52:46.516208 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:46.516186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg" event={"ID":"d4368c81-6749-45d2-b9d8-2912a0eaf827","Type":"ContainerDied","Data":"2d883ce29b1f8c01d6e1533d9195ad84238e0445ccd285b773388aaa18492edf"}
Jul 28 01:52:46.516676 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:46.516251    2577 scope.go:117] "RemoveContainer" containerID="e409043b4a02963e3b76bb042d4b79be89a5d483b8437acd98755f139eff6942"
Jul 28 01:52:46.539477 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:46.539452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"]
Jul 28 01:52:46.543274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:46.543253    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ffdmzg"]
Jul 28 01:52:47.698524 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:52:47.698490    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" path="/var/lib/kubelet/pods/d4368c81-6749-45d2-b9d8-2912a0eaf827/volumes"
Jul 28 01:53:28.944389 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.944351    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"]
Jul 28 01:53:28.944969 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.944951    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerName="main"
Jul 28 01:53:28.945030 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.944972    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerName="main"
Jul 28 01:53:28.945091 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.945080    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4368c81-6749-45d2-b9d8-2912a0eaf827" containerName="main"
Jul 28 01:53:28.948878 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.948843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:28.951592 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.951568    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 01:53:28.951726 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.951649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-5nqpm\""
Jul 28 01:53:28.951792 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.951772    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 01:53:28.952529 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.952513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 01:53:28.952623 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.952576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 01:53:28.959840 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:28.959816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"]
Jul 28 01:53:29.084916 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.084832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.084916 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.084900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffpc\" (UniqueName: \"kubernetes.io/projected/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kube-api-access-8ffpc\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.085123 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.084946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.085123 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.085011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.085123 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.085059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.085123 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.085082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.085259 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.085166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.176191 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.176152    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"]
Jul 28 01:53:29.179844 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.179820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.186578 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.186697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.186697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffpc\" (UniqueName: \"kubernetes.io/projected/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kube-api-access-8ffpc\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.186697 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.186909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.186909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.186909 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.186773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.187132 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.187099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.187236 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.187140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.187236 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.187216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.187372 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.187349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.189545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.189516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.189646 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.189563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.194077 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.194054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"]
Jul 28 01:53:29.198575 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.198529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffpc\" (UniqueName: \"kubernetes.io/projected/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kube-api-access-8ffpc\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.261538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.261513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:29.287992 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.287965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.288121 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.288101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.288167 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.288135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.389452 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.389415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.389607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.389501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.389607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.389519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.389825 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.389811    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.389944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.389921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.391964 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.391942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.405798 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.405773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"]
Jul 28 01:53:29.509585 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.509552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:53:29.665152 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.665119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"]
Jul 28 01:53:29.670109 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:53:29.670085    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972108e5_a2da_4d12_a8cc_4cb2ccef181e.slice/crio-d4ea17fd70f85fdde0b0f2a4d2872373d8314361eea11aa996a2b43706583214 WatchSource:0}: Error finding container d4ea17fd70f85fdde0b0f2a4d2872373d8314361eea11aa996a2b43706583214: Status 404 returned error can't find the container with id d4ea17fd70f85fdde0b0f2a4d2872373d8314361eea11aa996a2b43706583214
Jul 28 01:53:29.742539 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.742509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" event={"ID":"972108e5-a2da-4d12-a8cc-4cb2ccef181e","Type":"ContainerStarted","Data":"d4ea17fd70f85fdde0b0f2a4d2872373d8314361eea11aa996a2b43706583214"}
Jul 28 01:53:29.744597 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.744369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" event={"ID":"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4","Type":"ContainerStarted","Data":"e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e"}
Jul 28 01:53:29.744597 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:29.744422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" event={"ID":"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4","Type":"ContainerStarted","Data":"c61811fec14fa6aa7fc05a377c3fe7880370159ffb7e979a1017cd1c0d52cc1a"}
Jul 28 01:53:30.756249 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:30.756183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" event={"ID":"972108e5-a2da-4d12-a8cc-4cb2ccef181e","Type":"ContainerStarted","Data":"c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95"}
Jul 28 01:53:34.786379 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:34.786343    2577 generic.go:356] "Generic (PLEG): container finished" podID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerID="c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95" exitCode=0
Jul 28 01:53:34.786807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:34.786417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" event={"ID":"972108e5-a2da-4d12-a8cc-4cb2ccef181e","Type":"ContainerDied","Data":"c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95"}
Jul 28 01:53:34.788257 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:34.788233    2577 generic.go:356] "Generic (PLEG): container finished" podID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerID="e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e" exitCode=0
Jul 28 01:53:34.788357 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:34.788277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" event={"ID":"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4","Type":"ContainerDied","Data":"e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e"}
Jul 28 01:53:35.800301 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:35.800267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" event={"ID":"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4","Type":"ContainerStarted","Data":"96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"}
Jul 28 01:53:35.826913 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:35.826834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" podStartSLOduration=2.910124012 podStartE2EDuration="7.826812589s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.916688577s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:53:35.825749142 +0000 UTC m=+1666.660496688" watchObservedRunningTime="2026-07-28 01:53:35.826812589 +0000 UTC m=+1666.661560136"
Jul 28 01:53:36.262230 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:36.262197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:39.262334 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:39.262302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:39.278480 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:39.278452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:53:39.835933 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:53:39.835905    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:54:09.016139 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:09.016103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" event={"ID":"972108e5-a2da-4d12-a8cc-4cb2ccef181e","Type":"ContainerStarted","Data":"640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d"}
Jul 28 01:54:09.017163 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:09.017139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:54:09.035644 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:09.035593    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" podStartSLOduration=1.713622722 podStartE2EDuration="40.035578783s" totalImagesPullingTime="33.444950469s" totalInitContainerRuntime="4.877005592s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:54:09.034636303 +0000 UTC m=+1699.869383850" watchObservedRunningTime="2026-07-28 01:54:09.035578783 +0000 UTC m=+1699.870326329"
Jul 28 01:54:40.033447 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:40.033425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:54:41.482063 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.482037    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7_972108e5-a2da-4d12-a8cc-4cb2ccef181e/main/0.log"
Jul 28 01:54:41.746767 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.746680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"]
Jul 28 01:54:41.746951 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.746915    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerName="main" containerID="cri-o://640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d" gracePeriod=30
Jul 28 01:54:41.749388 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.749369    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"]
Jul 28 01:54:41.758501 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:54:41.758459    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:54:41.758501 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:54:41.758501 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:54:41.758501 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c" cmd=["/bin/sleep","15"]
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.758528    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" message="" err=<
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:54:41.758544    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="main" containerID="cri-o://96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"
Jul 28 01:54:41.758794 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.758598    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="main" containerID="cri-o://96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c" gracePeriod=60
Jul 28 01:54:41.825258 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:41.825221    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 28 01:54:42.159335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.159302    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:54:42.217677 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.217640    2577 generic.go:356] "Generic (PLEG): container finished" podID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerID="96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c" exitCode=0
Jul 28 01:54:42.217889 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.217794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" event={"ID":"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4","Type":"ContainerDied","Data":"96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"}
Jul 28 01:54:42.217889 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.217846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9" event={"ID":"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4","Type":"ContainerDied","Data":"c61811fec14fa6aa7fc05a377c3fe7880370159ffb7e979a1017cd1c0d52cc1a"}
Jul 28 01:54:42.218028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.217903    2577 scope.go:117] "RemoveContainer" containerID="96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"
Jul 28 01:54:42.218028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.217937    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"
Jul 28 01:54:42.272241 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.272212    2577 scope.go:117] "RemoveContainer" containerID="e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e"
Jul 28 01:54:42.286042 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.285759    2577 scope.go:117] "RemoveContainer" containerID="96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"
Jul 28 01:54:42.286522 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:54:42.286284    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c\": container with ID starting with 96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c not found: ID does not exist" containerID="96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"
Jul 28 01:54:42.286522 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.286322    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c"} err="failed to get container status \"96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c\": rpc error: code = NotFound desc = could not find container \"96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c\": container with ID starting with 96b3f02e86ba3ee4ad2f4e423c8b457958fde99c6a324ba7b76d47f1df93ef8c not found: ID does not exist"
Jul 28 01:54:42.286522 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.286348    2577 scope.go:117] "RemoveContainer" containerID="e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e"
Jul 28 01:54:42.286830 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:54:42.286800    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e\": container with ID starting with e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e not found: ID does not exist" containerID="e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e"
Jul 28 01:54:42.286941 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.286831    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e"} err="failed to get container status \"e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e\": rpc error: code = NotFound desc = could not find container \"e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e\": container with ID starting with e1db6512700d6bf95498d590c2c648b43f6e1720e12bfeb45ee950429829573e not found: ID does not exist"
Jul 28 01:54:42.313991 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.313916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tmp-dir\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314141 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.313991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-home\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314141 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-model-cache\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314141 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kserve-provision-location\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314141 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-dshm\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ffpc\" (UniqueName: \"kubernetes.io/projected/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kube-api-access-8ffpc\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.314348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tls-certs\") pod \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\" (UID: \"725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4\") "
Jul 28 01:54:42.314348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314264    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.314348 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-home" (OuterVolumeSpecName: "home") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.314604 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314510    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.314604 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314529    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.314604 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.314542    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.316988 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.316959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-dshm" (OuterVolumeSpecName: "dshm") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.317160 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.317069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:54:42.317349 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.317326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kube-api-access-8ffpc" (OuterVolumeSpecName: "kube-api-access-8ffpc") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "kube-api-access-8ffpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:54:42.387104 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.387062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" (UID: "725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.414506 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.414481    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:54:42.415026 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.415007    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.415026 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.415026    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.415161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.415036    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffpc\" (UniqueName: \"kubernetes.io/projected/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-kube-api-access-8ffpc\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.415161 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.415046    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.515586 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.515551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-kserve-provision-location\") pod \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") "
Jul 28 01:54:42.516047 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.515628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tls-certs\") pod \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") "
Jul 28 01:54:42.516047 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.515681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tmp-dir\") pod \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\" (UID: \"972108e5-a2da-4d12-a8cc-4cb2ccef181e\") "
Jul 28 01:54:42.516226 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.516200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "972108e5-a2da-4d12-a8cc-4cb2ccef181e" (UID: "972108e5-a2da-4d12-a8cc-4cb2ccef181e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.518176 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.518149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "972108e5-a2da-4d12-a8cc-4cb2ccef181e" (UID: "972108e5-a2da-4d12-a8cc-4cb2ccef181e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:54:42.548607 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.548577    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"]
Jul 28 01:54:42.556247 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.556213    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-fz2k9"]
Jul 28 01:54:42.585883 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.585786    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "972108e5-a2da-4d12-a8cc-4cb2ccef181e" (UID: "972108e5-a2da-4d12-a8cc-4cb2ccef181e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.617206 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.617177    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.617206 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.617202    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.617206 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:42.617210    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972108e5-a2da-4d12-a8cc-4cb2ccef181e-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:54:43.225755 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.225721    2577 generic.go:356] "Generic (PLEG): container finished" podID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerID="640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d" exitCode=0
Jul 28 01:54:43.225944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.225794    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"
Jul 28 01:54:43.225944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.225803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" event={"ID":"972108e5-a2da-4d12-a8cc-4cb2ccef181e","Type":"ContainerDied","Data":"640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d"}
Jul 28 01:54:43.225944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.225871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7" event={"ID":"972108e5-a2da-4d12-a8cc-4cb2ccef181e","Type":"ContainerDied","Data":"d4ea17fd70f85fdde0b0f2a4d2872373d8314361eea11aa996a2b43706583214"}
Jul 28 01:54:43.225944 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.225895    2577 scope.go:117] "RemoveContainer" containerID="640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d"
Jul 28 01:54:43.235526 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.235504    2577 scope.go:117] "RemoveContainer" containerID="c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95"
Jul 28 01:54:43.245026 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.245010    2577 scope.go:117] "RemoveContainer" containerID="640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d"
Jul 28 01:54:43.245292 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:54:43.245258    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d\": container with ID starting with 640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d not found: ID does not exist" containerID="640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d"
Jul 28 01:54:43.245390 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.245291    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d"} err="failed to get container status \"640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d\": rpc error: code = NotFound desc = could not find container \"640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d\": container with ID starting with 640e409ce19aaf0e8085caa509b22854899e06a1d338242bb5fa3f1bf9f6c47d not found: ID does not exist"
Jul 28 01:54:43.245390 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.245309    2577 scope.go:117] "RemoveContainer" containerID="c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95"
Jul 28 01:54:43.245705 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:54:43.245677    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95\": container with ID starting with c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95 not found: ID does not exist" containerID="c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95"
Jul 28 01:54:43.245798 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.245713    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95"} err="failed to get container status \"c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95\": rpc error: code = NotFound desc = could not find container \"c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95\": container with ID starting with c4c1e599b5ba682f378fec62e0d71ca02fa3b82e8de0f07224eb09399b091f95 not found: ID does not exist"
Jul 28 01:54:43.248577 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.248554    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"]
Jul 28 01:54:43.254829 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.254804    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-66pk7"]
Jul 28 01:54:43.697853 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.697759    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" path="/var/lib/kubelet/pods/725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4/volumes"
Jul 28 01:54:43.698403 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:54:43.698380    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" path="/var/lib/kubelet/pods/972108e5-a2da-4d12-a8cc-4cb2ccef181e/volumes"
Jul 28 01:55:09.747561 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.747524    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"]
Jul 28 01:55:09.748125 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748107    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerName="main"
Jul 28 01:55:09.748176 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748128    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerName="main"
Jul 28 01:55:09.748176 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748153    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerName="storage-initializer"
Jul 28 01:55:09.748176 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748162    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerName="storage-initializer"
Jul 28 01:55:09.748274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748182    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="storage-initializer"
Jul 28 01:55:09.748274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748192    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="storage-initializer"
Jul 28 01:55:09.748274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748202    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="main"
Jul 28 01:55:09.748274 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748211    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="main"
Jul 28 01:55:09.748393 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748298    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="972108e5-a2da-4d12-a8cc-4cb2ccef181e" containerName="main"
Jul 28 01:55:09.748393 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.748311    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="725b9ccf-d7d6-40e4-a6c8-4c41b01e9fb4" containerName="main"
Jul 28 01:55:09.751123 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.751103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.754559 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.754532    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 01:55:09.754559 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.754550    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 01:55:09.754742 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.754531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 01:55:09.754742 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.754594    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 01:55:09.754742 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.754532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-kqnzw\""
Jul 28 01:55:09.762668 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.762646    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"]
Jul 28 01:55:09.863028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.862998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.863028 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.863030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.863222 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.863049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.863222 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.863073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.863222 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.863150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.863222 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.863214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmj7\" (UniqueName: \"kubernetes.io/projected/6ea614bc-c91b-4033-9eae-b64fc06f9891-kube-api-access-4pmj7\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.863358 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.863249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea614bc-c91b-4033-9eae-b64fc06f9891-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964368 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmj7\" (UniqueName: \"kubernetes.io/projected/6ea614bc-c91b-4033-9eae-b64fc06f9891-kube-api-access-4pmj7\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea614bc-c91b-4033-9eae-b64fc06f9891-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964545 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964807 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.964932 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.965015 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.964988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.965073 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.965038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.965116 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.965072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.967100 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.967075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.967239 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.967220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea614bc-c91b-4033-9eae-b64fc06f9891-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:09.973077 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:09.973054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmj7\" (UniqueName: \"kubernetes.io/projected/6ea614bc-c91b-4033-9eae-b64fc06f9891-kube-api-access-4pmj7\") pod \"tokenizer-migration-path-test-kserve-569c688787-j7sdm\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:10.022954 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.022846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"]
Jul 28 01:55:10.026956 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.026934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.038311 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.038289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"]
Jul 28 01:55:10.063300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.063272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:10.065718 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.065346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b27b6dc-dd85-42e2-8402-4908a49de25d-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.065718 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.065456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.065718 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.065516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.166772 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.166742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.166935 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.166797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.166935 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.166887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b27b6dc-dd85-42e2-8402-4908a49de25d-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.167197 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.167174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.167897 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.167399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.170131 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.170099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b27b6dc-dd85-42e2-8402-4908a49de25d-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.208513 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.208485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"]
Jul 28 01:55:10.211332 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:55:10.211309    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea614bc_c91b_4033_9eae_b64fc06f9891.slice/crio-68a5da1150898808c8b4ecc502266997976db626bc99b53ea258837170b9e9e1 WatchSource:0}: Error finding container 68a5da1150898808c8b4ecc502266997976db626bc99b53ea258837170b9e9e1: Status 404 returned error can't find the container with id 68a5da1150898808c8b4ecc502266997976db626bc99b53ea258837170b9e9e1
Jul 28 01:55:10.338286 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.338202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:10.381705 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.381672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" event={"ID":"6ea614bc-c91b-4033-9eae-b64fc06f9891","Type":"ContainerStarted","Data":"4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5"}
Jul 28 01:55:10.381874 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.381728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" event={"ID":"6ea614bc-c91b-4033-9eae-b64fc06f9891","Type":"ContainerStarted","Data":"68a5da1150898808c8b4ecc502266997976db626bc99b53ea258837170b9e9e1"}
Jul 28 01:55:10.483846 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:10.483821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"]
Jul 28 01:55:10.486302 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:55:10.486274    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b27b6dc_dd85_42e2_8402_4908a49de25d.slice/crio-833fe44ae7cd31405340943fb63ff8405cad8a3bbcbc524bfa5651b56e533f5e WatchSource:0}: Error finding container 833fe44ae7cd31405340943fb63ff8405cad8a3bbcbc524bfa5651b56e533f5e: Status 404 returned error can't find the container with id 833fe44ae7cd31405340943fb63ff8405cad8a3bbcbc524bfa5651b56e533f5e
Jul 28 01:55:11.392975 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:11.392933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" event={"ID":"5b27b6dc-dd85-42e2-8402-4908a49de25d","Type":"ContainerStarted","Data":"80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b"}
Jul 28 01:55:11.393654 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:11.392998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" event={"ID":"5b27b6dc-dd85-42e2-8402-4908a49de25d","Type":"ContainerStarted","Data":"833fe44ae7cd31405340943fb63ff8405cad8a3bbcbc524bfa5651b56e533f5e"}
Jul 28 01:55:15.426087 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:15.426001    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerID="4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5" exitCode=0
Jul 28 01:55:15.426554 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:15.426081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" event={"ID":"6ea614bc-c91b-4033-9eae-b64fc06f9891","Type":"ContainerDied","Data":"4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5"}
Jul 28 01:55:15.427703 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:15.427681    2577 generic.go:356] "Generic (PLEG): container finished" podID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerID="80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b" exitCode=0
Jul 28 01:55:15.427797 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:15.427730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" event={"ID":"5b27b6dc-dd85-42e2-8402-4908a49de25d","Type":"ContainerDied","Data":"80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b"}
Jul 28 01:55:16.434785 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:16.434750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" event={"ID":"5b27b6dc-dd85-42e2-8402-4908a49de25d","Type":"ContainerStarted","Data":"ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e"}
Jul 28 01:55:16.435282 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:16.435000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:16.436709 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:16.436683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" event={"ID":"6ea614bc-c91b-4033-9eae-b64fc06f9891","Type":"ContainerStarted","Data":"6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"}
Jul 28 01:55:16.453845 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:16.453779    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" podStartSLOduration=2.893324115 podStartE2EDuration="7.453764798s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.560440683s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:16.451885843 +0000 UTC m=+1767.286633389" watchObservedRunningTime="2026-07-28 01:55:16.453764798 +0000 UTC m=+1767.288512343"
Jul 28 01:55:16.471885 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:16.471819    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" podStartSLOduration=2.641295554 podStartE2EDuration="7.471799803s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.830504249s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:16.469950191 +0000 UTC m=+1767.304697738" watchObservedRunningTime="2026-07-28 01:55:16.471799803 +0000 UTC m=+1767.306547350"
Jul 28 01:55:17.063941 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:17.063908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:20.064445 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:20.064404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:20.077151 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:20.077128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:20.470791 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:20.470768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:47.454325 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:47.454287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:48.319200 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:48.319169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg_5b27b6dc-dd85-42e2-8402-4908a49de25d/main/0.log"
Jul 28 01:55:48.782021 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:48.781989    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"]
Jul 28 01:55:48.782497 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:48.782223    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerName="main" containerID="cri-o://ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e" gracePeriod=30
Jul 28 01:55:48.798538 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:48.798506    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"]
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:55:48.809999    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262" cmd=["/bin/sleep","15"]
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:48.810057    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" message="" err=<
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:55:48.810068    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="main" containerID="cri-o://6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"
Jul 28 01:55:48.810179 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:48.810114    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="main" containerID="cri-o://6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262" gracePeriod=60
Jul 28 01:55:49.097959 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.097919    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:49.136563 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-home\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.136563 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-kserve-provision-location\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.136838 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-dshm\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.136838 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pmj7\" (UniqueName: \"kubernetes.io/projected/6ea614bc-c91b-4033-9eae-b64fc06f9891-kube-api-access-4pmj7\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.136838 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-model-cache\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.137032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea614bc-c91b-4033-9eae-b64fc06f9891-tls-certs\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.137032 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.136949    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-tmp-dir\") pod \"6ea614bc-c91b-4033-9eae-b64fc06f9891\" (UID: \"6ea614bc-c91b-4033-9eae-b64fc06f9891\") "
Jul 28 01:55:49.137816 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.137611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-home" (OuterVolumeSpecName: "home") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.137816 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.137624    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.138049 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.138014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.140579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.140452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-dshm" (OuterVolumeSpecName: "dshm") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.140579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.140500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea614bc-c91b-4033-9eae-b64fc06f9891-kube-api-access-4pmj7" (OuterVolumeSpecName: "kube-api-access-4pmj7") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "kube-api-access-4pmj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:55:49.140579 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.140535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea614bc-c91b-4033-9eae-b64fc06f9891-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:55:49.215332 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.215277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ea614bc-c91b-4033-9eae-b64fc06f9891" (UID: "6ea614bc-c91b-4033-9eae-b64fc06f9891"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.238427 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238407    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.238507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238430    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.238507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238445    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.238507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238456    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.238507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238465    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmj7\" (UniqueName: \"kubernetes.io/projected/6ea614bc-c91b-4033-9eae-b64fc06f9891-kube-api-access-4pmj7\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.238507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238474    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea614bc-c91b-4033-9eae-b64fc06f9891-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.238507 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.238485    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea614bc-c91b-4033-9eae-b64fc06f9891-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.428758 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.428723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:49.540960 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.540927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-kserve-provision-location\") pod \"5b27b6dc-dd85-42e2-8402-4908a49de25d\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") "
Jul 28 01:55:49.541155 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.540970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-tmp-dir\") pod \"5b27b6dc-dd85-42e2-8402-4908a49de25d\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") "
Jul 28 01:55:49.541155 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.541038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b27b6dc-dd85-42e2-8402-4908a49de25d-tls-certs\") pod \"5b27b6dc-dd85-42e2-8402-4908a49de25d\" (UID: \"5b27b6dc-dd85-42e2-8402-4908a49de25d\") "
Jul 28 01:55:49.541300 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.541275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b27b6dc-dd85-42e2-8402-4908a49de25d" (UID: "5b27b6dc-dd85-42e2-8402-4908a49de25d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.543374 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.543342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b27b6dc-dd85-42e2-8402-4908a49de25d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b27b6dc-dd85-42e2-8402-4908a49de25d" (UID: "5b27b6dc-dd85-42e2-8402-4908a49de25d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:55:49.601586 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.601506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b27b6dc-dd85-42e2-8402-4908a49de25d" (UID: "5b27b6dc-dd85-42e2-8402-4908a49de25d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.633169 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.633135    2577 generic.go:356] "Generic (PLEG): container finished" podID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerID="ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e" exitCode=0
Jul 28 01:55:49.633335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.633218    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"
Jul 28 01:55:49.633335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.633224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" event={"ID":"5b27b6dc-dd85-42e2-8402-4908a49de25d","Type":"ContainerDied","Data":"ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e"}
Jul 28 01:55:49.633335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.633309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg" event={"ID":"5b27b6dc-dd85-42e2-8402-4908a49de25d","Type":"ContainerDied","Data":"833fe44ae7cd31405340943fb63ff8405cad8a3bbcbc524bfa5651b56e533f5e"}
Jul 28 01:55:49.633493 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.633338    2577 scope.go:117] "RemoveContainer" containerID="ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e"
Jul 28 01:55:49.635041 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.635011    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerID="6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262" exitCode=0
Jul 28 01:55:49.635155 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.635075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" event={"ID":"6ea614bc-c91b-4033-9eae-b64fc06f9891","Type":"ContainerDied","Data":"6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"}
Jul 28 01:55:49.635155 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.635085    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"
Jul 28 01:55:49.635155 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.635109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm" event={"ID":"6ea614bc-c91b-4033-9eae-b64fc06f9891","Type":"ContainerDied","Data":"68a5da1150898808c8b4ecc502266997976db626bc99b53ea258837170b9e9e1"}
Jul 28 01:55:49.641699 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.641678    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.641784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.641716    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b27b6dc-dd85-42e2-8402-4908a49de25d-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.641784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.641728    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b27b6dc-dd85-42e2-8402-4908a49de25d-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.644014 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.643872    2577 scope.go:117] "RemoveContainer" containerID="80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b"
Jul 28 01:55:49.652646 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.652630    2577 scope.go:117] "RemoveContainer" containerID="ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e"
Jul 28 01:55:49.652900 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:55:49.652883    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e\": container with ID starting with ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e not found: ID does not exist" containerID="ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e"
Jul 28 01:55:49.652964 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.652908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e"} err="failed to get container status \"ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e\": rpc error: code = NotFound desc = could not find container \"ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e\": container with ID starting with ffd70b84c8815ead86a932329d9d9b7dda5cbc52a834ab52cf35bafab55ddc8e not found: ID does not exist"
Jul 28 01:55:49.652964 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.652925    2577 scope.go:117] "RemoveContainer" containerID="80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b"
Jul 28 01:55:49.653174 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:55:49.653154    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b\": container with ID starting with 80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b not found: ID does not exist" containerID="80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b"
Jul 28 01:55:49.653224 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.653180    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b"} err="failed to get container status \"80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b\": rpc error: code = NotFound desc = could not find container \"80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b\": container with ID starting with 80c21f37bb28153801c4406779bd068b4477a8ac6f2d56423a23d8ace4a5678b not found: ID does not exist"
Jul 28 01:55:49.653224 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.653197    2577 scope.go:117] "RemoveContainer" containerID="6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"
Jul 28 01:55:49.660425 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.660394    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"]
Jul 28 01:55:49.662074 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.662051    2577 scope.go:117] "RemoveContainer" containerID="4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5"
Jul 28 01:55:49.663014 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.662999    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-l6nwg"]
Jul 28 01:55:49.671293 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.671275    2577 scope.go:117] "RemoveContainer" containerID="6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"
Jul 28 01:55:49.671586 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:55:49.671567    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262\": container with ID starting with 6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262 not found: ID does not exist" containerID="6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"
Jul 28 01:55:49.671638 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.671593    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262"} err="failed to get container status \"6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262\": rpc error: code = NotFound desc = could not find container \"6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262\": container with ID starting with 6eb25e270d8eb0e67e162ea798b4cf4c68922b2a5d038a731a7ffb7655af1262 not found: ID does not exist"
Jul 28 01:55:49.671638 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.671611    2577 scope.go:117] "RemoveContainer" containerID="4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5"
Jul 28 01:55:49.671881 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:55:49.671837    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5\": container with ID starting with 4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5 not found: ID does not exist" containerID="4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5"
Jul 28 01:55:49.671978 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.671877    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5"} err="failed to get container status \"4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5\": rpc error: code = NotFound desc = could not find container \"4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5\": container with ID starting with 4827b391c102bb9b0d27c6efba075d58be00a7bc058716aed757ddcb68cd6fd5 not found: ID does not exist"
Jul 28 01:55:49.676983 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.676959    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"]
Jul 28 01:55:49.682783 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.682757    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-j7sdm"]
Jul 28 01:55:49.698628 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.698606    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" path="/var/lib/kubelet/pods/5b27b6dc-dd85-42e2-8402-4908a49de25d/volumes"
Jul 28 01:55:49.699088 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:49.699074    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" path="/var/lib/kubelet/pods/6ea614bc-c91b-4033-9eae-b64fc06f9891/volumes"
Jul 28 01:55:51.569480 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:51.569375    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:55:51.584024 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:55:51.578445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 01:56:20.055723 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.055688    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"]
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056126    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="main"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056140    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="main"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056149    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerName="main"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056155    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerName="main"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056163    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="storage-initializer"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056170    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="storage-initializer"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056176    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerName="storage-initializer"
Jul 28 01:56:20.056194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056181    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerName="storage-initializer"
Jul 28 01:56:20.056446 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056239    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b27b6dc-dd85-42e2-8402-4908a49de25d" containerName="main"
Jul 28 01:56:20.056446 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.056247    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea614bc-c91b-4033-9eae-b64fc06f9891" containerName="main"
Jul 28 01:56:20.058235 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.058216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.063026 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.063003    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 01:56:20.063150 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.063077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-gp4d9\""
Jul 28 01:56:20.063150 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.063120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 01:56:20.063258 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.063170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 01:56:20.073847 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.073821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"]
Jul 28 01:56:20.216307 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.216279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.216473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.216328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrbk\" (UniqueName: \"kubernetes.io/projected/829c2148-cb97-4f85-ba3f-ed27fd109d41-kube-api-access-ldrbk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.216473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.216389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c2148-cb97-4f85-ba3f-ed27fd109d41-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.216473 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.216448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.216580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.216479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.216580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.216538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.317948 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.317841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c2148-cb97-4f85-ba3f-ed27fd109d41-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318110 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.317952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318110 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.317986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318110 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.318007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318110 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.318038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318110 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.318076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrbk\" (UniqueName: \"kubernetes.io/projected/829c2148-cb97-4f85-ba3f-ed27fd109d41-kube-api-access-ldrbk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318486 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.318460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318616 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.318579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.318676 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.318653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.320486 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.320465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.320756 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.320729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c2148-cb97-4f85-ba3f-ed27fd109d41-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.334633 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.334603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrbk\" (UniqueName: \"kubernetes.io/projected/829c2148-cb97-4f85-ba3f-ed27fd109d41-kube-api-access-ldrbk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.368333 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.368313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:20.507660 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.507574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"]
Jul 28 01:56:20.510017 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:56:20.509987    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829c2148_cb97_4f85_ba3f_ed27fd109d41.slice/crio-41edac96b9e1cae8a98d14873dbada873ef56b625da16161c3604129e1193c53 WatchSource:0}: Error finding container 41edac96b9e1cae8a98d14873dbada873ef56b625da16161c3604129e1193c53: Status 404 returned error can't find the container with id 41edac96b9e1cae8a98d14873dbada873ef56b625da16161c3604129e1193c53
Jul 28 01:56:20.802098 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.802054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" event={"ID":"829c2148-cb97-4f85-ba3f-ed27fd109d41","Type":"ContainerStarted","Data":"d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"}
Jul 28 01:56:20.802284 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.802119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" event={"ID":"829c2148-cb97-4f85-ba3f-ed27fd109d41","Type":"ContainerStarted","Data":"41edac96b9e1cae8a98d14873dbada873ef56b625da16161c3604129e1193c53"}
Jul 28 01:56:20.821398 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:20.821302    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" podStartSLOduration=0.821283735 podStartE2EDuration="821.283735ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:20.820540126 +0000 UTC m=+1831.655287677" watchObservedRunningTime="2026-07-28 01:56:20.821283735 +0000 UTC m=+1831.656031280"
Jul 28 01:56:21.369027 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:21.368991    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:30.368779 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:30.368729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:30.384945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:30.384910    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:30.867916 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:30.867881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:53.991882 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:53.990406    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"]
Jul 28 01:56:54.018001 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:56:54.017956    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:56:54.018001 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:56:54.018001 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:56:54.018001 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6" cmd=["/bin/sleep","15"]
Jul 28 01:56:54.018400 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.018372    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" message="" err=<
Jul 28 01:56:54.018400 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:56:54.018400 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:56:54.018400 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:56:54.018660 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:56:54.018640    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:56:54.018660 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:56:54.018660 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:56:54.018660 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" podUID="829c2148-cb97-4f85-ba3f-ed27fd109d41" containerName="main" containerID="cri-o://d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"
Jul 28 01:56:54.018974 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.018952    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" podUID="829c2148-cb97-4f85-ba3f-ed27fd109d41" containerName="main" containerID="cri-o://d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6" gracePeriod=60
Jul 28 01:56:54.388072 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.388045    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:54.523688 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.523658    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-dshm\") pod \"829c2148-cb97-4f85-ba3f-ed27fd109d41\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") "
Jul 28 01:56:54.523688 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.523692    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-model-cache\") pod \"829c2148-cb97-4f85-ba3f-ed27fd109d41\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") "
Jul 28 01:56:54.523945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.523773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c2148-cb97-4f85-ba3f-ed27fd109d41-tls-certs\") pod \"829c2148-cb97-4f85-ba3f-ed27fd109d41\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") "
Jul 28 01:56:54.523945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.523801    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldrbk\" (UniqueName: \"kubernetes.io/projected/829c2148-cb97-4f85-ba3f-ed27fd109d41-kube-api-access-ldrbk\") pod \"829c2148-cb97-4f85-ba3f-ed27fd109d41\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") "
Jul 28 01:56:54.523945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.523826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-tmp-dir\") pod \"829c2148-cb97-4f85-ba3f-ed27fd109d41\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") "
Jul 28 01:56:54.523945 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.523896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-home\") pod \"829c2148-cb97-4f85-ba3f-ed27fd109d41\" (UID: \"829c2148-cb97-4f85-ba3f-ed27fd109d41\") "
Jul 28 01:56:54.524158 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.524088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "829c2148-cb97-4f85-ba3f-ed27fd109d41" (UID: "829c2148-cb97-4f85-ba3f-ed27fd109d41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:56:54.524158 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.524104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-model-cache" (OuterVolumeSpecName: "model-cache") pod "829c2148-cb97-4f85-ba3f-ed27fd109d41" (UID: "829c2148-cb97-4f85-ba3f-ed27fd109d41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:56:54.524280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.524159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-home" (OuterVolumeSpecName: "home") pod "829c2148-cb97-4f85-ba3f-ed27fd109d41" (UID: "829c2148-cb97-4f85-ba3f-ed27fd109d41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:56:54.524336 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.524284    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:56:54.524336 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.524304    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:56:54.524336 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.524319    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:56:54.526068 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.526035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829c2148-cb97-4f85-ba3f-ed27fd109d41-kube-api-access-ldrbk" (OuterVolumeSpecName: "kube-api-access-ldrbk") pod "829c2148-cb97-4f85-ba3f-ed27fd109d41" (UID: "829c2148-cb97-4f85-ba3f-ed27fd109d41"). InnerVolumeSpecName "kube-api-access-ldrbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:56:54.526068 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.526051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-dshm" (OuterVolumeSpecName: "dshm") pod "829c2148-cb97-4f85-ba3f-ed27fd109d41" (UID: "829c2148-cb97-4f85-ba3f-ed27fd109d41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:56:54.526218 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.526051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829c2148-cb97-4f85-ba3f-ed27fd109d41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "829c2148-cb97-4f85-ba3f-ed27fd109d41" (UID: "829c2148-cb97-4f85-ba3f-ed27fd109d41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:56:54.625510 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.625432    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c2148-cb97-4f85-ba3f-ed27fd109d41-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:56:54.625510 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.625460    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrbk\" (UniqueName: \"kubernetes.io/projected/829c2148-cb97-4f85-ba3f-ed27fd109d41-kube-api-access-ldrbk\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:56:54.625510 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:54.625472    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829c2148-cb97-4f85-ba3f-ed27fd109d41-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:56:55.014068 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.014033    2577 generic.go:356] "Generic (PLEG): container finished" podID="829c2148-cb97-4f85-ba3f-ed27fd109d41" containerID="d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6" exitCode=0
Jul 28 01:56:55.014518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.014074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"
Jul 28 01:56:55.014518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.014117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" event={"ID":"829c2148-cb97-4f85-ba3f-ed27fd109d41","Type":"ContainerDied","Data":"d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"}
Jul 28 01:56:55.014518 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.014179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5" event={"ID":"829c2148-cb97-4f85-ba3f-ed27fd109d41","Type":"ContainerDied","Data":"41edac96b9e1cae8a98d14873dbada873ef56b625da16161c3604129e1193c53"}
Jul 28 01:56:55.015543 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.014216    2577 scope.go:117] "RemoveContainer" containerID="d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"
Jul 28 01:56:55.025302 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.025110    2577 scope.go:117] "RemoveContainer" containerID="d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"
Jul 28 01:56:55.025379 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:56:55.025363    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6\": container with ID starting with d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6 not found: ID does not exist" containerID="d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"
Jul 28 01:56:55.025420 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.025386    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6"} err="failed to get container status \"d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6\": rpc error: code = NotFound desc = could not find container \"d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6\": container with ID starting with d09d877bed587f2c235d27e89957039ad59c3ac106d201b5d40040c15250cce6 not found: ID does not exist"
Jul 28 01:56:55.037056 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.037035    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"]
Jul 28 01:56:55.044048 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.044027    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-wg6p5"]
Jul 28 01:56:55.698132 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:56:55.698099    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="829c2148-cb97-4f85-ba3f-ed27fd109d41" path="/var/lib/kubelet/pods/829c2148-cb97-4f85-ba3f-ed27fd109d41/volumes"
Jul 28 01:57:24.241344 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.241269    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"]
Jul 28 01:57:24.241784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.241663    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829c2148-cb97-4f85-ba3f-ed27fd109d41" containerName="main"
Jul 28 01:57:24.241784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.241676    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="829c2148-cb97-4f85-ba3f-ed27fd109d41" containerName="main"
Jul 28 01:57:24.241784 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.241770    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="829c2148-cb97-4f85-ba3f-ed27fd109d41" containerName="main"
Jul 28 01:57:24.243896 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.243853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.247754 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.247733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 01:57:24.247886 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.247733    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 01:57:24.247886 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.247781    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-6js2f\""
Jul 28 01:57:24.247886 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.247764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 01:57:24.255080 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.255059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"]
Jul 28 01:57:24.290315 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.290286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z55k\" (UniqueName: \"kubernetes.io/projected/914357e8-3da9-4337-a5d6-6285c9d43458-kube-api-access-2z55k\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.290432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.290341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.290432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.290395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914357e8-3da9-4337-a5d6-6285c9d43458-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.290432 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.290414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.290542 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.290502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.290580 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.290540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391542 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391542 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z55k\" (UniqueName: \"kubernetes.io/projected/914357e8-3da9-4337-a5d6-6285c9d43458-kube-api-access-2z55k\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914357e8-3da9-4337-a5d6-6285c9d43458-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391761 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.391995 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.391970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.392070 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.392020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.392070 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.392061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.393961 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.393930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.394188 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.394168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914357e8-3da9-4337-a5d6-6285c9d43458-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.399280 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.399264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z55k\" (UniqueName: \"kubernetes.io/projected/914357e8-3da9-4337-a5d6-6285c9d43458-kube-api-access-2z55k\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.554570 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.554485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:24.723693 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:57:24.723657    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914357e8_3da9_4337_a5d6_6285c9d43458.slice/crio-c16300b75ab70c9c839b54492d783edb6b4716e45a055da228ccd6d740d0649a WatchSource:0}: Error finding container c16300b75ab70c9c839b54492d783edb6b4716e45a055da228ccd6d740d0649a: Status 404 returned error can't find the container with id c16300b75ab70c9c839b54492d783edb6b4716e45a055da228ccd6d740d0649a
Jul 28 01:57:24.725577 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:24.725555    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"]
Jul 28 01:57:25.190319 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:25.190247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" event={"ID":"914357e8-3da9-4337-a5d6-6285c9d43458","Type":"ContainerStarted","Data":"b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a"}
Jul 28 01:57:25.190532 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:25.190345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" event={"ID":"914357e8-3da9-4337-a5d6-6285c9d43458","Type":"ContainerStarted","Data":"c16300b75ab70c9c839b54492d783edb6b4716e45a055da228ccd6d740d0649a"}
Jul 28 01:57:25.213222 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:25.213171    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" podStartSLOduration=1.21315296 podStartE2EDuration="1.21315296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:57:25.211637066 +0000 UTC m=+1896.046384611" watchObservedRunningTime="2026-07-28 01:57:25.21315296 +0000 UTC m=+1896.047900505"
Jul 28 01:57:25.555418 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:25.555380    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:34.554654 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:34.554620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:34.567013 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:34.566986    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:57:35.255076 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:57:35.255045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:59:05.435536 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.435456    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"]
Jul 28 01:59:05.445318 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:59:05.445281    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:59:05.445318 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:59:05.445318 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:59:05.445318 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a" cmd=["/bin/sleep","15"]
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.445336    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" message="" err=<
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]: E0728 01:59:05.445349    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" podUID="914357e8-3da9-4337-a5d6-6285c9d43458" containerName="main" containerID="cri-o://b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a"
Jul 28 01:59:05.445539 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.445386    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" podUID="914357e8-3da9-4337-a5d6-6285c9d43458" containerName="main" containerID="cri-o://b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a" gracePeriod=60
Jul 28 01:59:05.750626 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.750591    2577 generic.go:356] "Generic (PLEG): container finished" podID="914357e8-3da9-4337-a5d6-6285c9d43458" containerID="b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a" exitCode=0
Jul 28 01:59:05.750797 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.750650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" event={"ID":"914357e8-3da9-4337-a5d6-6285c9d43458","Type":"ContainerDied","Data":"b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a"}
Jul 28 01:59:05.807462 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.807439    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:59:05.908888 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.908837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-home\") pod \"914357e8-3da9-4337-a5d6-6285c9d43458\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") "
Jul 28 01:59:05.909059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.908935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z55k\" (UniqueName: \"kubernetes.io/projected/914357e8-3da9-4337-a5d6-6285c9d43458-kube-api-access-2z55k\") pod \"914357e8-3da9-4337-a5d6-6285c9d43458\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") "
Jul 28 01:59:05.909059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.908962    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-tmp-dir\") pod \"914357e8-3da9-4337-a5d6-6285c9d43458\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") "
Jul 28 01:59:05.909059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.908981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914357e8-3da9-4337-a5d6-6285c9d43458-tls-certs\") pod \"914357e8-3da9-4337-a5d6-6285c9d43458\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") "
Jul 28 01:59:05.909059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-model-cache\") pod \"914357e8-3da9-4337-a5d6-6285c9d43458\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") "
Jul 28 01:59:05.909059 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909021    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-dshm\") pod \"914357e8-3da9-4337-a5d6-6285c9d43458\" (UID: \"914357e8-3da9-4337-a5d6-6285c9d43458\") "
Jul 28 01:59:05.909338 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-home" (OuterVolumeSpecName: "home") pod "914357e8-3da9-4337-a5d6-6285c9d43458" (UID: "914357e8-3da9-4337-a5d6-6285c9d43458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:59:05.909338 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "914357e8-3da9-4337-a5d6-6285c9d43458" (UID: "914357e8-3da9-4337-a5d6-6285c9d43458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:59:05.909338 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909289    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-model-cache" (OuterVolumeSpecName: "model-cache") pod "914357e8-3da9-4337-a5d6-6285c9d43458" (UID: "914357e8-3da9-4337-a5d6-6285c9d43458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:59:05.909498 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909346    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:59:05.909498 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.909366    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:59:05.911272 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.911244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914357e8-3da9-4337-a5d6-6285c9d43458-kube-api-access-2z55k" (OuterVolumeSpecName: "kube-api-access-2z55k") pod "914357e8-3da9-4337-a5d6-6285c9d43458" (UID: "914357e8-3da9-4337-a5d6-6285c9d43458"). InnerVolumeSpecName "kube-api-access-2z55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:59:05.911272 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.911245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914357e8-3da9-4337-a5d6-6285c9d43458-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "914357e8-3da9-4337-a5d6-6285c9d43458" (UID: "914357e8-3da9-4337-a5d6-6285c9d43458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:59:05.911428 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:05.911316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-dshm" (OuterVolumeSpecName: "dshm") pod "914357e8-3da9-4337-a5d6-6285c9d43458" (UID: "914357e8-3da9-4337-a5d6-6285c9d43458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:59:06.009813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.009784    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914357e8-3da9-4337-a5d6-6285c9d43458-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:59:06.009813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.009810    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:59:06.009813 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.009818    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914357e8-3da9-4337-a5d6-6285c9d43458-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:59:06.010063 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.009827    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z55k\" (UniqueName: \"kubernetes.io/projected/914357e8-3da9-4337-a5d6-6285c9d43458-kube-api-access-2z55k\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 01:59:06.757292 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.757260    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"
Jul 28 01:59:06.757716 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.757300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w" event={"ID":"914357e8-3da9-4337-a5d6-6285c9d43458","Type":"ContainerDied","Data":"c16300b75ab70c9c839b54492d783edb6b4716e45a055da228ccd6d740d0649a"}
Jul 28 01:59:06.757716 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.757363    2577 scope.go:117] "RemoveContainer" containerID="b94031464a1d307962bfc45aee6410dfdccc2e0390c10312cbb058de03049d7a"
Jul 28 01:59:06.790106 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.790073    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"]
Jul 28 01:59:06.796528 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:06.796506    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-q8r8w"]
Jul 28 01:59:07.698335 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:07.698298    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="914357e8-3da9-4337-a5d6-6285c9d43458" path="/var/lib/kubelet/pods/914357e8-3da9-4337-a5d6-6285c9d43458/volumes"
Jul 28 01:59:41.733590 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.733550    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"]
Jul 28 01:59:41.734439 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.734411    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914357e8-3da9-4337-a5d6-6285c9d43458" containerName="main"
Jul 28 01:59:41.734439 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.734441    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="914357e8-3da9-4337-a5d6-6285c9d43458" containerName="main"
Jul 28 01:59:41.734687 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.734651    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="914357e8-3da9-4337-a5d6-6285c9d43458" containerName="main"
Jul 28 01:59:41.738407 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.738383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.742494 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.742470    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 01:59:41.742494 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.742483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-pgwkj\""
Jul 28 01:59:41.742686 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.742506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 01:59:41.742686 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.742545    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 01:59:41.745051 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.745028    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"]
Jul 28 01:59:41.827620 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.827589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.827790 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.827625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7j5\" (UniqueName: \"kubernetes.io/projected/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-kube-api-access-9m7j5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.827790 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.827668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.827790 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.827748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.827790 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.827779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.828005 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.827853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.928811 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.928778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.928811 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.928815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.928890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929108 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.929093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929226 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.929132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7j5\" (UniqueName: \"kubernetes.io/projected/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-kube-api-access-9m7j5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929226 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.929201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929347 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.929312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929347 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.929337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.929536 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.929519    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.931096 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.931070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.931462 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.931445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:41.936912 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:41.936890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7j5\" (UniqueName: \"kubernetes.io/projected/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-kube-api-access-9m7j5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:42.051459 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:42.051371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:42.194352 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:42.194327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"]
Jul 28 01:59:42.196303 ip-10-0-138-218 kubenswrapper[2577]: W0728 01:59:42.196276    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94852ac_b72f_4a9f_8e27_a8bb089c6a36.slice/crio-89a28612f4840032f4204f0af1e97fd6753f15766434a3f12efa840b6aa8e342 WatchSource:0}: Error finding container 89a28612f4840032f4204f0af1e97fd6753f15766434a3f12efa840b6aa8e342: Status 404 returned error can't find the container with id 89a28612f4840032f4204f0af1e97fd6753f15766434a3f12efa840b6aa8e342
Jul 28 01:59:42.957755 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:42.957722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" event={"ID":"a94852ac-b72f-4a9f-8e27-a8bb089c6a36","Type":"ContainerStarted","Data":"369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"}
Jul 28 01:59:42.958149 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:42.957767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" event={"ID":"a94852ac-b72f-4a9f-8e27-a8bb089c6a36","Type":"ContainerStarted","Data":"89a28612f4840032f4204f0af1e97fd6753f15766434a3f12efa840b6aa8e342"}
Jul 28 01:59:42.983876 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:42.983815    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" podStartSLOduration=1.983800595 podStartE2EDuration="1.983800595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:42.981214216 +0000 UTC m=+2033.815961762" watchObservedRunningTime="2026-07-28 01:59:42.983800595 +0000 UTC m=+2033.818548140"
Jul 28 01:59:43.051955 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:43.051925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:52.052194 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:52.052157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:52.064670 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:52.064645    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 01:59:53.024780 ip-10-0-138-218 kubenswrapper[2577]: I0728 01:59:53.024739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 02:00:51.610313 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:00:51.610198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:00:51.617466 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:00:51.617445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:01:18.319597 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.319556    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"]
Jul 28 02:01:18.329237 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:01:18.329163    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:01:18.329237 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:01:18.329237 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 02:01:18.329237 ip-10-0-138-218 kubenswrapper[2577]:  > containerID="369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4" cmd=["/bin/sleep","15"]
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.329253    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" message="" err=<
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]:  >
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:01:18.329269    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" podUID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" containerName="main" containerID="cri-o://369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"
Jul 28 02:01:18.329491 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.329308    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" podUID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" containerName="main" containerID="cri-o://369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4" gracePeriod=60
Jul 28 02:01:18.687302 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.687270    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 02:01:18.800628 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-model-cache\") pod \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") "
Jul 28 02:01:18.800628 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tmp-dir\") pod \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") "
Jul 28 02:01:18.800928 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800690    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-dshm\") pod \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") "
Jul 28 02:01:18.800928 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800732    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-home\") pod \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") "
Jul 28 02:01:18.800928 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tls-certs\") pod \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") "
Jul 28 02:01:18.800928 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m7j5\" (UniqueName: \"kubernetes.io/projected/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-kube-api-access-9m7j5\") pod \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\" (UID: \"a94852ac-b72f-4a9f-8e27-a8bb089c6a36\") "
Jul 28 02:01:18.801154 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.800978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-model-cache" (OuterVolumeSpecName: "model-cache") pod "a94852ac-b72f-4a9f-8e27-a8bb089c6a36" (UID: "a94852ac-b72f-4a9f-8e27-a8bb089c6a36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:01:18.801154 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.801067    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a94852ac-b72f-4a9f-8e27-a8bb089c6a36" (UID: "a94852ac-b72f-4a9f-8e27-a8bb089c6a36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:01:18.801240 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.801144    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-home" (OuterVolumeSpecName: "home") pod "a94852ac-b72f-4a9f-8e27-a8bb089c6a36" (UID: "a94852ac-b72f-4a9f-8e27-a8bb089c6a36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:01:18.801304 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.801252    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:01:18.801304 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.801289    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:01:18.801304 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.801298    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:01:18.803183 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.803152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-dshm" (OuterVolumeSpecName: "dshm") pod "a94852ac-b72f-4a9f-8e27-a8bb089c6a36" (UID: "a94852ac-b72f-4a9f-8e27-a8bb089c6a36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:01:18.803311 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.803207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a94852ac-b72f-4a9f-8e27-a8bb089c6a36" (UID: "a94852ac-b72f-4a9f-8e27-a8bb089c6a36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:01:18.803311 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.803270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-kube-api-access-9m7j5" (OuterVolumeSpecName: "kube-api-access-9m7j5") pod "a94852ac-b72f-4a9f-8e27-a8bb089c6a36" (UID: "a94852ac-b72f-4a9f-8e27-a8bb089c6a36"). InnerVolumeSpecName "kube-api-access-9m7j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:01:18.902415 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.902317    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:01:18.902415 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.902362    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m7j5\" (UniqueName: \"kubernetes.io/projected/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-kube-api-access-9m7j5\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:01:18.902415 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:18.902377    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94852ac-b72f-4a9f-8e27-a8bb089c6a36-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:01:19.487715 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.487683    2577 generic.go:356] "Generic (PLEG): container finished" podID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" containerID="369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4" exitCode=0
Jul 28 02:01:19.488137 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.487746    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"
Jul 28 02:01:19.488137 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.487763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" event={"ID":"a94852ac-b72f-4a9f-8e27-a8bb089c6a36","Type":"ContainerDied","Data":"369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"}
Jul 28 02:01:19.488137 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.487846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc" event={"ID":"a94852ac-b72f-4a9f-8e27-a8bb089c6a36","Type":"ContainerDied","Data":"89a28612f4840032f4204f0af1e97fd6753f15766434a3f12efa840b6aa8e342"}
Jul 28 02:01:19.488137 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.487898    2577 scope.go:117] "RemoveContainer" containerID="369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"
Jul 28 02:01:19.498669 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.498652    2577 scope.go:117] "RemoveContainer" containerID="369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"
Jul 28 02:01:19.498922 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:01:19.498901    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4\": container with ID starting with 369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4 not found: ID does not exist" containerID="369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"
Jul 28 02:01:19.498993 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.498934    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4"} err="failed to get container status \"369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4\": rpc error: code = NotFound desc = could not find container \"369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4\": container with ID starting with 369ac3ed07d856001d767efbb0f0db58a9fb0fe162a836d175c7324e43cb63e4 not found: ID does not exist"
Jul 28 02:01:19.512009 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.511986    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"]
Jul 28 02:01:19.515270 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.515246    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679698wwxc"]
Jul 28 02:01:19.697893 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:01:19.697846    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" path="/var/lib/kubelet/pods/a94852ac-b72f-4a9f-8e27-a8bb089c6a36/volumes"
Jul 28 02:02:03.626884 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.626817    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"]
Jul 28 02:02:03.627673 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.627242    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" containerName="main"
Jul 28 02:02:03.627673 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.627260    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" containerName="main"
Jul 28 02:02:03.627673 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.627356    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94852ac-b72f-4a9f-8e27-a8bb089c6a36" containerName="main"
Jul 28 02:02:03.630592 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.630575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.633349 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.633323    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 28 02:02:03.633349 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.633340    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 02:02:03.633573 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.633396    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 02:02:03.633573 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.633399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-4q6jr\""
Jul 28 02:02:03.633573 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.633327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 02:02:03.633573 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.633432    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:02:03.642442 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.642418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"]
Jul 28 02:02:03.682397 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eed48b-91fa-4e16-ad86-016f7c1f5828-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682529 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11eed48b-91fa-4e16-ad86-016f7c1f5828-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682529 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682529 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682529 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682691 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682691 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.682691 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.682582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pd6s\" (UniqueName: \"kubernetes.io/projected/11eed48b-91fa-4e16-ad86-016f7c1f5828-kube-api-access-5pd6s\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.783679 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11eed48b-91fa-4e16-ad86-016f7c1f5828-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.783833 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.783833 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.783833 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.783833 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.783833 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784096 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pd6s\" (UniqueName: \"kubernetes.io/projected/11eed48b-91fa-4e16-ad86-016f7c1f5828-kube-api-access-5pd6s\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784096 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.783983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eed48b-91fa-4e16-ad86-016f7c1f5828-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784199 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.784120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784199 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.784173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784307 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.784272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784427 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.784409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.784486 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.784437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11eed48b-91fa-4e16-ad86-016f7c1f5828-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.786185 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.786158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.786345 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.786328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eed48b-91fa-4e16-ad86-016f7c1f5828-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.792168 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.792151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pd6s\" (UniqueName: \"kubernetes.io/projected/11eed48b-91fa-4e16-ad86-016f7c1f5828-kube-api-access-5pd6s\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:03.942324 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:03.942241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:04.090149 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:04.090119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"]
Jul 28 02:02:04.092284 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:02:04.092240    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11eed48b_91fa_4e16_ad86_016f7c1f5828.slice/crio-4d92bbd702637094fecfeacf37eb33b5040904cdf6cf03a94a828fe1510e18e2 WatchSource:0}: Error finding container 4d92bbd702637094fecfeacf37eb33b5040904cdf6cf03a94a828fe1510e18e2: Status 404 returned error can't find the container with id 4d92bbd702637094fecfeacf37eb33b5040904cdf6cf03a94a828fe1510e18e2
Jul 28 02:02:04.736592 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:04.736556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" event={"ID":"11eed48b-91fa-4e16-ad86-016f7c1f5828","Type":"ContainerStarted","Data":"a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395"}
Jul 28 02:02:04.736980 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:04.736614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" event={"ID":"11eed48b-91fa-4e16-ad86-016f7c1f5828","Type":"ContainerStarted","Data":"4d92bbd702637094fecfeacf37eb33b5040904cdf6cf03a94a828fe1510e18e2"}
Jul 28 02:02:09.764512 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:09.764478    2577 generic.go:356] "Generic (PLEG): container finished" podID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerID="a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395" exitCode=0
Jul 28 02:02:09.764902 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:09.764552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" event={"ID":"11eed48b-91fa-4e16-ad86-016f7c1f5828","Type":"ContainerDied","Data":"a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395"}
Jul 28 02:02:10.774206 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:10.774172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" event={"ID":"11eed48b-91fa-4e16-ad86-016f7c1f5828","Type":"ContainerStarted","Data":"53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a"}
Jul 28 02:02:10.799025 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:10.798973    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podStartSLOduration=2.701930373 podStartE2EDuration="7.798958217s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.097027844s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:02:10.79738285 +0000 UTC m=+2181.632130397" watchObservedRunningTime="2026-07-28 02:02:10.798958217 +0000 UTC m=+2181.633705762"
Jul 28 02:02:10.942891 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:10.942832    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:13.942896 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:13.942839    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:02:13.944616 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:13.944587    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:02:23.943498 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:23.943440    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:02:33.943344 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:33.943296    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:02:43.943210 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:43.943169    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:02:53.943580 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:02:53.943532    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:03:03.943208 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:03:03.943170    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:03:13.943199 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:03:13.943108    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:03:23.943174 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:03:23.943120    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 28 02:03:33.953160 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:03:33.953123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:03:33.960988 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:03:33.960968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:03:58.975567 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:03:58.975530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"]
Jul 28 02:04:14.007521 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.007463    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main" containerID="cri-o://53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a" gracePeriod=45
Jul 28 02:04:14.662650 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.662625    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:04:14.712203 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eed48b-91fa-4e16-ad86-016f7c1f5828-tls-certs\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712203 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-dshm\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712440 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-kserve-provision-location\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712440 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pd6s\" (UniqueName: \"kubernetes.io/projected/11eed48b-91fa-4e16-ad86-016f7c1f5828-kube-api-access-5pd6s\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712440 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712299    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-model-cache\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712440 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-home\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712440 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712380    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11eed48b-91fa-4e16-ad86-016f7c1f5828-cabundle-cert\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712440 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-tmp-dir\") pod \"11eed48b-91fa-4e16-ad86-016f7c1f5828\" (UID: \"11eed48b-91fa-4e16-ad86-016f7c1f5828\") "
Jul 28 02:04:14.712766 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-model-cache" (OuterVolumeSpecName: "model-cache") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:04:14.712766 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712753    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.712901 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.712848    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-home" (OuterVolumeSpecName: "home") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:04:14.713112 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.713087    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11eed48b-91fa-4e16-ad86-016f7c1f5828-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:04:14.714587 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.714558    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-dshm" (OuterVolumeSpecName: "dshm") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:04:14.714821 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.714795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eed48b-91fa-4e16-ad86-016f7c1f5828-kube-api-access-5pd6s" (OuterVolumeSpecName: "kube-api-access-5pd6s") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "kube-api-access-5pd6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:04:14.715106 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.715056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11eed48b-91fa-4e16-ad86-016f7c1f5828-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:04:14.729764 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.729733    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:04:14.775992 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.775960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11eed48b-91fa-4e16-ad86-016f7c1f5828" (UID: "11eed48b-91fa-4e16-ad86-016f7c1f5828"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:04:14.813637 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813613    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.813637 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813638    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11eed48b-91fa-4e16-ad86-016f7c1f5828-cabundle-cert\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.813785 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813649    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.813785 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813658    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eed48b-91fa-4e16-ad86-016f7c1f5828-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.813785 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813665    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.813785 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813675    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11eed48b-91fa-4e16-ad86-016f7c1f5828-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:14.813785 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:14.813683    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pd6s\" (UniqueName: \"kubernetes.io/projected/11eed48b-91fa-4e16-ad86-016f7c1f5828-kube-api-access-5pd6s\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:04:15.525434 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.525401    2577 generic.go:356] "Generic (PLEG): container finished" podID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerID="53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a" exitCode=0
Jul 28 02:04:15.525852 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.525453    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"
Jul 28 02:04:15.525852 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.525488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" event={"ID":"11eed48b-91fa-4e16-ad86-016f7c1f5828","Type":"ContainerDied","Data":"53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a"}
Jul 28 02:04:15.525852 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.525545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp" event={"ID":"11eed48b-91fa-4e16-ad86-016f7c1f5828","Type":"ContainerDied","Data":"4d92bbd702637094fecfeacf37eb33b5040904cdf6cf03a94a828fe1510e18e2"}
Jul 28 02:04:15.525852 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.525569    2577 scope.go:117] "RemoveContainer" containerID="53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a"
Jul 28 02:04:15.535394 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.535376    2577 scope.go:117] "RemoveContainer" containerID="a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395"
Jul 28 02:04:15.543774 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.543758    2577 scope.go:117] "RemoveContainer" containerID="53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a"
Jul 28 02:04:15.544385 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:04:15.544368    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a\": container with ID starting with 53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a not found: ID does not exist" containerID="53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a"
Jul 28 02:04:15.544432 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.544392    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a"} err="failed to get container status \"53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a\": rpc error: code = NotFound desc = could not find container \"53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a\": container with ID starting with 53b0bac688f3af72025b27f1c3a37018cdd1bddb0ab1ac1596344818c640e21a not found: ID does not exist"
Jul 28 02:04:15.544432 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.544411    2577 scope.go:117] "RemoveContainer" containerID="a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395"
Jul 28 02:04:15.544622 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:04:15.544608    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395\": container with ID starting with a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395 not found: ID does not exist" containerID="a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395"
Jul 28 02:04:15.544658 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.544626    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395"} err="failed to get container status \"a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395\": rpc error: code = NotFound desc = could not find container \"a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395\": container with ID starting with a68846633efb9df03daed9fcd8dedd8ce93b5b62b34051ba832c934a6b6f8395 not found: ID does not exist"
Jul 28 02:04:15.548925 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.548900    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"]
Jul 28 02:04:15.552521 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.552499    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-9wlhp"]
Jul 28 02:04:15.697921 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:15.697890    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" path="/var/lib/kubelet/pods/11eed48b-91fa-4e16-ad86-016f7c1f5828/volumes"
Jul 28 02:04:38.723435 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.723399    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"]
Jul 28 02:04:38.724022 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.724003    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main"
Jul 28 02:04:38.724067 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.724024    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main"
Jul 28 02:04:38.724067 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.724060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="storage-initializer"
Jul 28 02:04:38.724136 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.724069    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="storage-initializer"
Jul 28 02:04:38.724171 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.724146    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11eed48b-91fa-4e16-ad86-016f7c1f5828" containerName="main"
Jul 28 02:04:38.727334 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.727319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.730140 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.730114    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 02:04:38.730262 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.730149    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:04:38.730262 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.730154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 02:04:38.730262 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.730227    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 28 02:04:38.730262 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.730244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 02:04:38.730511 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.730495    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-hf85n\""
Jul 28 02:04:38.737665 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.737644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"]
Jul 28 02:04:38.841316 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h66s\" (UniqueName: \"kubernetes.io/projected/eacfa995-0a23-4d26-b20f-d51fd3161440-kube-api-access-4h66s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841483 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841483 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841483 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eacfa995-0a23-4d26-b20f-d51fd3161440-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841483 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841624 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacfa995-0a23-4d26-b20f-d51fd3161440-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841624 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.841624 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.841540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942024 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.941994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h66s\" (UniqueName: \"kubernetes.io/projected/eacfa995-0a23-4d26-b20f-d51fd3161440-kube-api-access-4h66s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942024 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942283 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942283 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eacfa995-0a23-4d26-b20f-d51fd3161440-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942283 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942283 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacfa995-0a23-4d26-b20f-d51fd3161440-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942283 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942283 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942560 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942669 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942739 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942739 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.942919 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.942897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eacfa995-0a23-4d26-b20f-d51fd3161440-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.944616 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.944593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.945155 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.945137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacfa995-0a23-4d26-b20f-d51fd3161440-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:38.949579 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:38.949554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h66s\" (UniqueName: \"kubernetes.io/projected/eacfa995-0a23-4d26-b20f-d51fd3161440-kube-api-access-4h66s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:39.038592 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:39.038502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:39.181834 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:39.181810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"]
Jul 28 02:04:39.184018 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:04:39.183990    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacfa995_0a23_4d26_b20f_d51fd3161440.slice/crio-550f04663305d7a03d1e55c4a847d5fd032ea013ba2063bf6a8b603a946024dc WatchSource:0}: Error finding container 550f04663305d7a03d1e55c4a847d5fd032ea013ba2063bf6a8b603a946024dc: Status 404 returned error can't find the container with id 550f04663305d7a03d1e55c4a847d5fd032ea013ba2063bf6a8b603a946024dc
Jul 28 02:04:39.660289 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:39.660251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" event={"ID":"eacfa995-0a23-4d26-b20f-d51fd3161440","Type":"ContainerStarted","Data":"b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a"}
Jul 28 02:04:39.660453 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:39.660313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" event={"ID":"eacfa995-0a23-4d26-b20f-d51fd3161440","Type":"ContainerStarted","Data":"550f04663305d7a03d1e55c4a847d5fd032ea013ba2063bf6a8b603a946024dc"}
Jul 28 02:04:45.700830 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:45.700676    2577 generic.go:356] "Generic (PLEG): container finished" podID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerID="b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a" exitCode=0
Jul 28 02:04:45.700830 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:45.700750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" event={"ID":"eacfa995-0a23-4d26-b20f-d51fd3161440","Type":"ContainerDied","Data":"b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a"}
Jul 28 02:04:46.708012 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:46.707970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" event={"ID":"eacfa995-0a23-4d26-b20f-d51fd3161440","Type":"ContainerStarted","Data":"c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18"}
Jul 28 02:04:46.733769 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:46.733715    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podStartSLOduration=2.687114872 podStartE2EDuration="8.733699277s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.046584405s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:04:46.732132674 +0000 UTC m=+2337.566880220" watchObservedRunningTime="2026-07-28 02:04:46.733699277 +0000 UTC m=+2337.568446842"
Jul 28 02:04:47.038848 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:47.038814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:49.039491 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:49.039452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:04:49.041096 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:49.041062    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:04:59.039302 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:04:59.039263    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:05:09.039252 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:09.039211    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:05:19.039484 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:19.039437    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:05:29.039595 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:29.039556    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:05:39.038878 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:39.038818    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:05:49.039174 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:49.039135    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:05:51.650267 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:51.650160    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:05:51.657907 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:51.657884    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:05:59.038965 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:05:59.038926    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 28 02:06:09.049708 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:09.049673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:06:09.058643 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:09.058612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:06:11.236997 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:11.236915    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"]
Jul 28 02:06:26.259623 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.259556    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main" containerID="cri-o://c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18" gracePeriod=45
Jul 28 02:06:26.813544 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.813515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:06:26.829807 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.829779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-tmp-dir\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.829951 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.829847    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-model-cache\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.829951 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.829899    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-home\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.830080 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.829953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eacfa995-0a23-4d26-b20f-d51fd3161440-cabundle-cert\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.830080 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830012    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h66s\" (UniqueName: \"kubernetes.io/projected/eacfa995-0a23-4d26-b20f-d51fd3161440-kube-api-access-4h66s\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.830080 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacfa995-0a23-4d26-b20f-d51fd3161440-tls-certs\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.830080 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-kserve-provision-location\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.830267 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-model-cache" (OuterVolumeSpecName: "model-cache") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:06:26.830267 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-dshm\") pod \"eacfa995-0a23-4d26-b20f-d51fd3161440\" (UID: \"eacfa995-0a23-4d26-b20f-d51fd3161440\") "
Jul 28 02:06:26.830267 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:06:26.830413 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-home" (OuterVolumeSpecName: "home") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:06:26.830413 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacfa995-0a23-4d26-b20f-d51fd3161440-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:06:26.830413 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830407    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.830565 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.830427    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.832271 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.832236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacfa995-0a23-4d26-b20f-d51fd3161440-kube-api-access-4h66s" (OuterVolumeSpecName: "kube-api-access-4h66s") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "kube-api-access-4h66s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:06:26.832271 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.832248    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-dshm" (OuterVolumeSpecName: "dshm") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:06:26.832466 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.832319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacfa995-0a23-4d26-b20f-d51fd3161440-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:06:26.898020 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.897939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eacfa995-0a23-4d26-b20f-d51fd3161440" (UID: "eacfa995-0a23-4d26-b20f-d51fd3161440"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:06:26.930808 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.930785    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-kserve-provision-location\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.930808 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.930807    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.931008 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.930817    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eacfa995-0a23-4d26-b20f-d51fd3161440-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.931008 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.930825    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eacfa995-0a23-4d26-b20f-d51fd3161440-cabundle-cert\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.931008 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.930836    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h66s\" (UniqueName: \"kubernetes.io/projected/eacfa995-0a23-4d26-b20f-d51fd3161440-kube-api-access-4h66s\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:26.931008 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:26.930844    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eacfa995-0a23-4d26-b20f-d51fd3161440-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:27.288710 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.288677    2577 generic.go:356] "Generic (PLEG): container finished" podID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerID="c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18" exitCode=0
Jul 28 02:06:27.289169 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.288727    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"
Jul 28 02:06:27.289169 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.288765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" event={"ID":"eacfa995-0a23-4d26-b20f-d51fd3161440","Type":"ContainerDied","Data":"c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18"}
Jul 28 02:06:27.289169 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.288817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7" event={"ID":"eacfa995-0a23-4d26-b20f-d51fd3161440","Type":"ContainerDied","Data":"550f04663305d7a03d1e55c4a847d5fd032ea013ba2063bf6a8b603a946024dc"}
Jul 28 02:06:27.289169 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.288840    2577 scope.go:117] "RemoveContainer" containerID="c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18"
Jul 28 02:06:27.298775 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.298760    2577 scope.go:117] "RemoveContainer" containerID="b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a"
Jul 28 02:06:27.307665 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.307649    2577 scope.go:117] "RemoveContainer" containerID="c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18"
Jul 28 02:06:27.307922 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:06:27.307903    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18\": container with ID starting with c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18 not found: ID does not exist" containerID="c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18"
Jul 28 02:06:27.307979 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.307931    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18"} err="failed to get container status \"c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18\": rpc error: code = NotFound desc = could not find container \"c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18\": container with ID starting with c6c8ef47a279f23fb286485ab80a6149640dbe205f1b2421cae868260380ad18 not found: ID does not exist"
Jul 28 02:06:27.307979 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.307948    2577 scope.go:117] "RemoveContainer" containerID="b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a"
Jul 28 02:06:27.308160 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:06:27.308144    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a\": container with ID starting with b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a not found: ID does not exist" containerID="b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a"
Jul 28 02:06:27.308207 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.308164    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a"} err="failed to get container status \"b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a\": rpc error: code = NotFound desc = could not find container \"b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a\": container with ID starting with b5978fa15637e8ae8d52ea3c3045e45214042c472f1f214a262722a417d98d0a not found: ID does not exist"
Jul 28 02:06:27.312114 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.312093    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"]
Jul 28 02:06:27.317875 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.317840    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-8zcb7"]
Jul 28 02:06:27.697906 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:27.697809    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" path="/var/lib/kubelet/pods/eacfa995-0a23-4d26-b20f-d51fd3161440/volumes"
Jul 28 02:06:34.532477 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.532444    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"]
Jul 28 02:06:34.532846 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.532810    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main"
Jul 28 02:06:34.532846 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.532822    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main"
Jul 28 02:06:34.532846 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.532843    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="storage-initializer"
Jul 28 02:06:34.532846 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.532849    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="storage-initializer"
Jul 28 02:06:34.532999 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.532931    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eacfa995-0a23-4d26-b20f-d51fd3161440" containerName="main"
Jul 28 02:06:34.535990 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.535974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:34.539554 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.539532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-dpvh8\""
Jul 28 02:06:34.540291 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.540273    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 02:06:34.540627 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.540610    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 02:06:34.552992 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.552970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"]
Jul 28 02:06:34.599348 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.599321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kgg\" (UniqueName: \"kubernetes.io/projected/afa33f78-a20b-4ce4-a3e3-a1b200e17e32-kube-api-access-r6kgg\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:34.700385 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.700357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kgg\" (UniqueName: \"kubernetes.io/projected/afa33f78-a20b-4ce4-a3e3-a1b200e17e32-kube-api-access-r6kgg\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:34.713336 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:34.713307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kgg\" (UniqueName: \"kubernetes.io/projected/afa33f78-a20b-4ce4-a3e3-a1b200e17e32-kube-api-access-r6kgg\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.137409 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.137372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.142968 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.142948    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.237979 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.237945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.238098 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.238076    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.241238 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.241221    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") DevicePath \"csi-ab964df41ec5083d915397af9caaef9df07a7508d079646d30c902021724de24\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.458706 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.458672    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/95b27c29700c5ae5452bc2c09ed9eefdb49ca1fa064bdcd99631c6795f6fd554/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.473559 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.473532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"e2e-pvc-model-download-2v5n4\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.645446 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.645411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:39.778450 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:39.778426    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"]
Jul 28 02:06:39.780244 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:06:39.780219    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa33f78_a20b_4ce4_a3e3_a1b200e17e32.slice/crio-f8c63528ccd5957f4e540f888bdc1890b4f52b482eb532ff61e325f67ec6c9c1 WatchSource:0}: Error finding container f8c63528ccd5957f4e540f888bdc1890b4f52b482eb532ff61e325f67ec6c9c1: Status 404 returned error can't find the container with id f8c63528ccd5957f4e540f888bdc1890b4f52b482eb532ff61e325f67ec6c9c1
Jul 28 02:06:40.364595 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:40.364506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4" event={"ID":"afa33f78-a20b-4ce4-a3e3-a1b200e17e32","Type":"ContainerStarted","Data":"bf95616fd1eb5b26e7ded0557714ab1b8b7a5378c82d40372d124cee61ba577d"}
Jul 28 02:06:40.365148 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:40.364813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4" event={"ID":"afa33f78-a20b-4ce4-a3e3-a1b200e17e32","Type":"ContainerStarted","Data":"f8c63528ccd5957f4e540f888bdc1890b4f52b482eb532ff61e325f67ec6c9c1"}
Jul 28 02:06:40.379732 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:40.379667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4" podStartSLOduration=9.379652635 podStartE2EDuration="9.379652635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:06:40.379170554 +0000 UTC m=+2451.213918100" watchObservedRunningTime="2026-07-28 02:06:40.379652635 +0000 UTC m=+2451.214400182"
Jul 28 02:06:46.403918 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:46.403886    2577 generic.go:356] "Generic (PLEG): container finished" podID="afa33f78-a20b-4ce4-a3e3-a1b200e17e32" containerID="bf95616fd1eb5b26e7ded0557714ab1b8b7a5378c82d40372d124cee61ba577d" exitCode=0
Jul 28 02:06:46.404260 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:46.403952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4" event={"ID":"afa33f78-a20b-4ce4-a3e3-a1b200e17e32","Type":"ContainerDied","Data":"bf95616fd1eb5b26e7ded0557714ab1b8b7a5378c82d40372d124cee61ba577d"}
Jul 28 02:06:47.588338 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.588316    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:47.614588 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.614563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") "
Jul 28 02:06:47.614742 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.614646    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6kgg\" (UniqueName: \"kubernetes.io/projected/afa33f78-a20b-4ce4-a3e3-a1b200e17e32-kube-api-access-r6kgg\") pod \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\" (UID: \"afa33f78-a20b-4ce4-a3e3-a1b200e17e32\") "
Jul 28 02:06:47.617159 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.617122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa33f78-a20b-4ce4-a3e3-a1b200e17e32-kube-api-access-r6kgg" (OuterVolumeSpecName: "kube-api-access-r6kgg") pod "afa33f78-a20b-4ce4-a3e3-a1b200e17e32" (UID: "afa33f78-a20b-4ce4-a3e3-a1b200e17e32"). InnerVolumeSpecName "kube-api-access-r6kgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:06:47.619086 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.619065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2" (OuterVolumeSpecName: "model-storage") pod "afa33f78-a20b-4ce4-a3e3-a1b200e17e32" (UID: "afa33f78-a20b-4ce4-a3e3-a1b200e17e32"). InnerVolumeSpecName "pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 02:06:47.715820 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.715795    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") on node \"ip-10-0-138-218.ec2.internal\" "
Jul 28 02:06:47.715820 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.715821    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kgg\" (UniqueName: \"kubernetes.io/projected/afa33f78-a20b-4ce4-a3e3-a1b200e17e32-kube-api-access-r6kgg\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:06:47.830986 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.830966    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2") on node "ip-10-0-138-218.ec2.internal"
Jul 28 02:06:47.917665 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:47.917635    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"csi-ab964df41ec5083d915397af9caaef9df07a7508d079646d30c902021724de24\""
Jul 28 02:06:48.417246 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:48.417218    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"
Jul 28 02:06:48.417416 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:48.417269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4" event={"ID":"afa33f78-a20b-4ce4-a3e3-a1b200e17e32","Type":"ContainerDied","Data":"f8c63528ccd5957f4e540f888bdc1890b4f52b482eb532ff61e325f67ec6c9c1"}
Jul 28 02:06:48.417416 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:48.417305    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c63528ccd5957f4e540f888bdc1890b4f52b482eb532ff61e325f67ec6c9c1"
Jul 28 02:06:56.714124 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.714089    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"]
Jul 28 02:06:56.714741 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.714714    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa33f78-a20b-4ce4-a3e3-a1b200e17e32" containerName="storage-initializer"
Jul 28 02:06:56.714947 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.714924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa33f78-a20b-4ce4-a3e3-a1b200e17e32" containerName="storage-initializer"
Jul 28 02:06:56.715137 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.715121    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afa33f78-a20b-4ce4-a3e3-a1b200e17e32" containerName="storage-initializer"
Jul 28 02:06:56.718157 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.718138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.720965 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.720947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 02:06:56.721085 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.720965    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 02:06:56.721085 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.721039    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 02:06:56.721085 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.721078    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-dpvh8\""
Jul 28 02:06:56.728528 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.728505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"]
Jul 28 02:06:56.800187 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.800157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsxv\" (UniqueName: \"kubernetes.io/projected/47987bc4-60ee-426d-aa8e-9eec0ea5689e-kube-api-access-ltsxv\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.800343 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.800192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.800343 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.800311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.800470 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.800360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.800470 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.800389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.800470 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.800438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901260 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901260 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901490 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901490 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901490 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsxv\" (UniqueName: \"kubernetes.io/projected/47987bc4-60ee-426d-aa8e-9eec0ea5689e-kube-api-access-ltsxv\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901490 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901715 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.901837 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.902030 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.901997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.903536 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.903513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.903970 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.903947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:56.910780 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:56.910746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsxv\" (UniqueName: \"kubernetes.io/projected/47987bc4-60ee-426d-aa8e-9eec0ea5689e-kube-api-access-ltsxv\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.422360 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.422327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.434505 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.434474    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.523324 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.523292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.523476 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.523441    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.526221 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.526201    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") DevicePath \"csi-ab964df41ec5083d915397af9caaef9df07a7508d079646d30c902021724de24\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.614936 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.614905    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/95b27c29700c5ae5452bc2c09ed9eefdb49ca1fa064bdcd99631c6795f6fd554/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.632594 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.632568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.730462 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.730425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:06:59.867681 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:06:59.867651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"]
Jul 28 02:06:59.871435 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:06:59.871404    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47987bc4_60ee_426d_aa8e_9eec0ea5689e.slice/crio-c0050408a7f0501721a3e307b1958a6222ef270a7951c0970187c9352da904d3 WatchSource:0}: Error finding container c0050408a7f0501721a3e307b1958a6222ef270a7951c0970187c9352da904d3: Status 404 returned error can't find the container with id c0050408a7f0501721a3e307b1958a6222ef270a7951c0970187c9352da904d3
Jul 28 02:07:00.483624 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:00.483589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" event={"ID":"47987bc4-60ee-426d-aa8e-9eec0ea5689e","Type":"ContainerStarted","Data":"95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba"}
Jul 28 02:07:00.484104 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:00.483652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" event={"ID":"47987bc4-60ee-426d-aa8e-9eec0ea5689e","Type":"ContainerStarted","Data":"c0050408a7f0501721a3e307b1958a6222ef270a7951c0970187c9352da904d3"}
Jul 28 02:07:00.504311 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:00.504234    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podStartSLOduration=4.504217067 podStartE2EDuration="4.504217067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:07:00.504098959 +0000 UTC m=+2471.338846504" watchObservedRunningTime="2026-07-28 02:07:00.504217067 +0000 UTC m=+2471.338964635"
Jul 28 02:07:00.731383 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:00.731349    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:07:09.731201 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:09.731169    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:07:09.732937 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:09.732907    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:07:19.731040 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:19.730995    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:07:29.731002 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:29.730956    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:07:39.730972 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:39.730933    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:07:49.731805 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:49.731712    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:07:59.731543 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:07:59.731500    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:08:09.731113 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:09.731070    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:08:19.731839 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:19.731787    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:08:29.731178 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:29.731134    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:08:39.731045 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:39.730992    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:08:49.730930 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:49.730891    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:08:59.741374 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:59.741338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:08:59.748939 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:08:59.748915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:09:04.618233 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:04.618195    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"]
Jul 28 02:09:19.641668 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:19.641583    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" containerID="cri-o://95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba" gracePeriod=45
Jul 28 02:09:19.741879 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:19.741831    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 28 02:09:20.303695 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.303673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:09:20.308077 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.308053    2577 generic.go:356] "Generic (PLEG): container finished" podID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerID="95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba" exitCode=0
Jul 28 02:09:20.308190 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.308088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"
Jul 28 02:09:20.308190 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.308127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" event={"ID":"47987bc4-60ee-426d-aa8e-9eec0ea5689e","Type":"ContainerDied","Data":"95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba"}
Jul 28 02:09:20.308190 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.308177    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx" event={"ID":"47987bc4-60ee-426d-aa8e-9eec0ea5689e","Type":"ContainerDied","Data":"c0050408a7f0501721a3e307b1958a6222ef270a7951c0970187c9352da904d3"}
Jul 28 02:09:20.308298 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.308198    2577 scope.go:117] "RemoveContainer" containerID="95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba"
Jul 28 02:09:20.316892 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.316874    2577 scope.go:117] "RemoveContainer" containerID="95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba"
Jul 28 02:09:20.317144 ip-10-0-138-218 kubenswrapper[2577]: E0728 02:09:20.317127    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba\": container with ID starting with 95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba not found: ID does not exist" containerID="95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba"
Jul 28 02:09:20.317190 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.317152    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba"} err="failed to get container status \"95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba\": rpc error: code = NotFound desc = could not find container \"95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba\": container with ID starting with 95e3c246b51310ca49b34ba469277abfe3e1679ee15eb3454308247c553befba not found: ID does not exist"
Jul 28 02:09:20.395355 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-home\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395355 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltsxv\" (UniqueName: \"kubernetes.io/projected/47987bc4-60ee-426d-aa8e-9eec0ea5689e-kube-api-access-ltsxv\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395355 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-dshm\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395576 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tmp-dir\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395576 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tls-certs\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395576 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-model-cache\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395686 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395579    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") pod \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\" (UID: \"47987bc4-60ee-426d-aa8e-9eec0ea5689e\") "
Jul 28 02:09:20.395991 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.395965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-model-cache" (OuterVolumeSpecName: "model-cache") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:09:20.396111 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.396047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-home" (OuterVolumeSpecName: "home") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:09:20.398113 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.398075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-dshm" (OuterVolumeSpecName: "dshm") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:09:20.398221 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.398117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47987bc4-60ee-426d-aa8e-9eec0ea5689e-kube-api-access-ltsxv" (OuterVolumeSpecName: "kube-api-access-ltsxv") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "kube-api-access-ltsxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:09:20.398221 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.398164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:09:20.398598 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.398579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2" (OuterVolumeSpecName: "kserve-pvc-source") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 02:09:20.409261 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.409228    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47987bc4-60ee-426d-aa8e-9eec0ea5689e" (UID: "47987bc4-60ee-426d-aa8e-9eec0ea5689e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:09:20.496316 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496286    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-model-cache\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.496469 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496328    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") on node \"ip-10-0-138-218.ec2.internal\" "
Jul 28 02:09:20.496469 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496339    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-home\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.496469 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496349    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltsxv\" (UniqueName: \"kubernetes.io/projected/47987bc4-60ee-426d-aa8e-9eec0ea5689e-kube-api-access-ltsxv\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.496469 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496358    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-dshm\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.496469 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496367    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tmp-dir\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.496469 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.496374    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47987bc4-60ee-426d-aa8e-9eec0ea5689e-tls-certs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:20.526494 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.526473    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2") on node "ip-10-0-138-218.ec2.internal"
Jul 28 02:09:20.596758 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.596721    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-5d76bca5-7a49-45c2-a8aa-68e33a057900\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05cc7d6070823cbe2\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"csi-ab964df41ec5083d915397af9caaef9df07a7508d079646d30c902021724de24\""
Jul 28 02:09:20.638191 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.638160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"]
Jul 28 02:09:20.643034 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:20.643009    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcgnltx"]
Jul 28 02:09:21.698774 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:21.698738    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" path="/var/lib/kubelet/pods/47987bc4-60ee-426d-aa8e-9eec0ea5689e/volumes"
Jul 28 02:09:27.852464 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.852432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"]
Jul 28 02:09:27.852952 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.852806    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main"
Jul 28 02:09:27.852952 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.852818    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main"
Jul 28 02:09:27.852952 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.852916    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="47987bc4-60ee-426d-aa8e-9eec0ea5689e" containerName="main"
Jul 28 02:09:27.855642 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.855627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:27.858477 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.858453    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 02:09:27.858616 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.858505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-ws6bz\""
Jul 28 02:09:27.859391 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.859374    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 02:09:27.863330 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.863310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"]
Jul 28 02:09:27.955233 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:27.955208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2rs\" (UniqueName: \"kubernetes.io/projected/384c047b-4ab6-4674-a809-34a97c742f61-kube-api-access-rn2rs\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:28.056030 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:28.056002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2rs\" (UniqueName: \"kubernetes.io/projected/384c047b-4ab6-4674-a809-34a97c742f61-kube-api-access-rn2rs\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:28.065491 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:28.065462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2rs\" (UniqueName: \"kubernetes.io/projected/384c047b-4ab6-4674-a809-34a97c742f61-kube-api-access-rn2rs\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:29.522702 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:29.522668    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"]
Jul 28 02:09:29.526704 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:29.526677    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2v5n4"]
Jul 28 02:09:29.699194 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:29.699153    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afa33f78-a20b-4ce4-a3e3-a1b200e17e32" path="/var/lib/kubelet/pods/afa33f78-a20b-4ce4-a3e3-a1b200e17e32/volumes"
Jul 28 02:09:32.901536 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:32.901508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:32.908357 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:32.908332    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.002071 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.002048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.002286 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.002230    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.005588 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.005572    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") DevicePath \"csi-60a7998b0b9225adb45f1053afefcc4775b1d675bd21632128d1190e277d8b07\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.224792 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.224759    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/208a350f6e40bdd253e6082b869e435ad86991889384313a6e3a0863d77206c7/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.242611 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.242586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"e2e-pvc-model-download-cpzzq\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.265552 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.265526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:33.399019 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:33.398987    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"]
Jul 28 02:09:33.400462 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:09:33.400440    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod384c047b_4ab6_4674_a809_34a97c742f61.slice/crio-0c4c0cfb57f8f58fcb140cbe15e9f9b409324a1e61699df65a68bc08799ba526 WatchSource:0}: Error finding container 0c4c0cfb57f8f58fcb140cbe15e9f9b409324a1e61699df65a68bc08799ba526: Status 404 returned error can't find the container with id 0c4c0cfb57f8f58fcb140cbe15e9f9b409324a1e61699df65a68bc08799ba526
Jul 28 02:09:34.394748 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:34.394691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq" event={"ID":"384c047b-4ab6-4674-a809-34a97c742f61","Type":"ContainerStarted","Data":"885143e508a059cb552bd7fd6f0d9b6c636852f2c14dc2720a8d0d59c7887577"}
Jul 28 02:09:34.395257 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:34.394757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq" event={"ID":"384c047b-4ab6-4674-a809-34a97c742f61","Type":"ContainerStarted","Data":"0c4c0cfb57f8f58fcb140cbe15e9f9b409324a1e61699df65a68bc08799ba526"}
Jul 28 02:09:34.415222 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:34.415156    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq" podStartSLOduration=10.415094425 podStartE2EDuration="10.415094425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:09:34.414437676 +0000 UTC m=+2625.249185222" watchObservedRunningTime="2026-07-28 02:09:34.415094425 +0000 UTC m=+2625.249841963"
Jul 28 02:09:40.432473 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:40.432392    2577 generic.go:356] "Generic (PLEG): container finished" podID="384c047b-4ab6-4674-a809-34a97c742f61" containerID="885143e508a059cb552bd7fd6f0d9b6c636852f2c14dc2720a8d0d59c7887577" exitCode=0
Jul 28 02:09:40.432473 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:40.432444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq" event={"ID":"384c047b-4ab6-4674-a809-34a97c742f61","Type":"ContainerDied","Data":"885143e508a059cb552bd7fd6f0d9b6c636852f2c14dc2720a8d0d59c7887577"}
Jul 28 02:09:41.589681 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.589648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:41.682162 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.682131    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn2rs\" (UniqueName: \"kubernetes.io/projected/384c047b-4ab6-4674-a809-34a97c742f61-kube-api-access-rn2rs\") pod \"384c047b-4ab6-4674-a809-34a97c742f61\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") "
Jul 28 02:09:41.682339 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.682266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"384c047b-4ab6-4674-a809-34a97c742f61\" (UID: \"384c047b-4ab6-4674-a809-34a97c742f61\") "
Jul 28 02:09:41.684388 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.684361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384c047b-4ab6-4674-a809-34a97c742f61-kube-api-access-rn2rs" (OuterVolumeSpecName: "kube-api-access-rn2rs") pod "384c047b-4ab6-4674-a809-34a97c742f61" (UID: "384c047b-4ab6-4674-a809-34a97c742f61"). InnerVolumeSpecName "kube-api-access-rn2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:09:41.685100 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.685078    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406" (OuterVolumeSpecName: "model-storage") pod "384c047b-4ab6-4674-a809-34a97c742f61" (UID: "384c047b-4ab6-4674-a809-34a97c742f61"). InnerVolumeSpecName "pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 02:09:41.784002 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.783973    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn2rs\" (UniqueName: \"kubernetes.io/projected/384c047b-4ab6-4674-a809-34a97c742f61-kube-api-access-rn2rs\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:09:41.784131 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.784015    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") on node \"ip-10-0-138-218.ec2.internal\" "
Jul 28 02:09:41.904207 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.904183    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406") on node "ip-10-0-138-218.ec2.internal"
Jul 28 02:09:41.985792 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:41.985763    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"csi-60a7998b0b9225adb45f1053afefcc4775b1d675bd21632128d1190e277d8b07\""
Jul 28 02:09:42.444516 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:42.444437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq"
Jul 28 02:09:42.444516 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:42.444455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq" event={"ID":"384c047b-4ab6-4674-a809-34a97c742f61","Type":"ContainerDied","Data":"0c4c0cfb57f8f58fcb140cbe15e9f9b409324a1e61699df65a68bc08799ba526"}
Jul 28 02:09:42.444516 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:42.444486    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c4c0cfb57f8f58fcb140cbe15e9f9b409324a1e61699df65a68bc08799ba526"
Jul 28 02:09:55.680490 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.680451    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"]
Jul 28 02:09:55.680910 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.680823    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384c047b-4ab6-4674-a809-34a97c742f61" containerName="storage-initializer"
Jul 28 02:09:55.680910 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.680834    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="384c047b-4ab6-4674-a809-34a97c742f61" containerName="storage-initializer"
Jul 28 02:09:55.680997 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.680946    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="384c047b-4ab6-4674-a809-34a97c742f61" containerName="storage-initializer"
Jul 28 02:09:55.685491 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.685463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.688160 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.688135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 02:09:55.688579 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.688557    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 02:09:55.688701 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.688611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-ws6bz\""
Jul 28 02:09:55.689146 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.689118    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-d6c9z\""
Jul 28 02:09:55.689269 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.689204    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 02:09:55.705186 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.705155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"]
Jul 28 02:09:55.805882 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.805831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.805882 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.805873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958b6fc7-dfbc-4f43-9da6-3bdb212df050-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.806090 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.805947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.806090 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.805980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.806090 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.806038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.806090 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.806070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9c6w\" (UniqueName: \"kubernetes.io/projected/958b6fc7-dfbc-4f43-9da6-3bdb212df050-kube-api-access-c9c6w\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907230 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907411 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958b6fc7-dfbc-4f43-9da6-3bdb212df050-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907411 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907411 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907411 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907630 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9c6w\" (UniqueName: \"kubernetes.io/projected/958b6fc7-dfbc-4f43-9da6-3bdb212df050-kube-api-access-c9c6w\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907707 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907767 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.907812 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.907768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.909475 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.909458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958b6fc7-dfbc-4f43-9da6-3bdb212df050-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.909951 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.909930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958b6fc7-dfbc-4f43-9da6-3bdb212df050-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:09:55.915423 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:09:55.915392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9c6w\" (UniqueName: \"kubernetes.io/projected/958b6fc7-dfbc-4f43-9da6-3bdb212df050-kube-api-access-c9c6w\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.477879 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.477831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.487516 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.487484    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.578356 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.578329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.578509 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.578488    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.581788 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.581770    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") DevicePath \"csi-60a7998b0b9225adb45f1053afefcc4775b1d675bd21632128d1190e277d8b07\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.671466 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.671426    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/208a350f6e40bdd253e6082b869e435ad86991889384313a6e3a0863d77206c7/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.687498 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.687472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08d597410d2aea406\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs\" (UID: \"958b6fc7-dfbc-4f43-9da6-3bdb212df050\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.797493 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.797405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:03.945686 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:03.945664    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"]
Jul 28 02:10:03.947443 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:10:03.947416    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958b6fc7_dfbc_4f43_9da6_3bdb212df050.slice/crio-8edeb30e26a4b8a7ca8e4c2b04729d11f2ca4b13ce92511c58f72b47b6ccc331 WatchSource:0}: Error finding container 8edeb30e26a4b8a7ca8e4c2b04729d11f2ca4b13ce92511c58f72b47b6ccc331: Status 404 returned error can't find the container with id 8edeb30e26a4b8a7ca8e4c2b04729d11f2ca4b13ce92511c58f72b47b6ccc331
Jul 28 02:10:04.568927 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:04.568890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" event={"ID":"958b6fc7-dfbc-4f43-9da6-3bdb212df050","Type":"ContainerStarted","Data":"b59926b6c6a031cda0e5e61e8bf7cd7de8ab9bd666a5e8989c9c0c1735026b05"}
Jul 28 02:10:04.569327 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:04.568959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:04.569327 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:04.568986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" event={"ID":"958b6fc7-dfbc-4f43-9da6-3bdb212df050","Type":"ContainerStarted","Data":"8edeb30e26a4b8a7ca8e4c2b04729d11f2ca4b13ce92511c58f72b47b6ccc331"}
Jul 28 02:10:05.577056 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:05.577022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" event={"ID":"958b6fc7-dfbc-4f43-9da6-3bdb212df050","Type":"ContainerStarted","Data":"5190846334f6b35205ccf198245c8c6c5dbebe27060dddb8bb4f20aef5887a70"}
Jul 28 02:10:05.601132 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:05.601079    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podStartSLOduration=10.60106312 podStartE2EDuration="10.60106312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:10:05.599328437 +0000 UTC m=+2656.434075983" watchObservedRunningTime="2026-07-28 02:10:05.60106312 +0000 UTC m=+2656.435810676"
Jul 28 02:10:13.798005 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:13.797964    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:13.798543 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:13.798039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:13.799756 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:13.799721    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:10:23.797937 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:23.797857    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:10:23.810132 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:23.810104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:10:33.798645 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:33.798598    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:10:43.798106 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:43.798001    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:10:51.690477 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:51.690371    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:10:51.699972 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:51.699950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:10:53.798897 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:10:53.798828    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:11:03.798295 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:11:03.798237    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:11:13.798811 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:11:13.798746    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:11:23.798669 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:11:23.798620    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:11:33.798295 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:11:33.798243    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:11:43.798737 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:11:43.798677    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:11:53.798509 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:11:53.798458    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs" podUID="958b6fc7-dfbc-4f43-9da6-3bdb212df050" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 28 02:12:03.816575 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:12:03.816543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:12:03.828475 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:12:03.828449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs"
Jul 28 02:12:50.274012 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:12:50.273980    2577 scope.go:117] "RemoveContainer" containerID="bf95616fd1eb5b26e7ded0557714ab1b8b7a5378c82d40372d124cee61ba577d"
Jul 28 02:15:51.727509 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:15:51.727486    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:15:51.746154 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:15:51.746137    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:20:51.767608 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:20:51.767502    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:20:51.791267 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:20:51.791242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:24:45.713271 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:45.713189    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs_958b6fc7-dfbc-4f43-9da6-3bdb212df050/llm-d-routing-sidecar/0.log"
Jul 28 02:24:45.723483 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:45.723454    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs_958b6fc7-dfbc-4f43-9da6-3bdb212df050/main/0.log"
Jul 28 02:24:50.569224 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.569185    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"]
Jul 28 02:24:50.572818 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.572798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:50.575202 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.575133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 02:24:50.575202 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.575177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5jrnk\""
Jul 28 02:24:50.575202 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.575189    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 02:24:50.586343 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.586318    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"]
Jul 28 02:24:50.660885 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.660819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8vz\" (UniqueName: \"kubernetes.io/projected/0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b-kube-api-access-6m8vz\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:50.762454 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.762412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8vz\" (UniqueName: \"kubernetes.io/projected/0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b-kube-api-access-6m8vz\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:50.770453 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:50.770420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8vz\" (UniqueName: \"kubernetes.io/projected/0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b-kube-api-access-6m8vz\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:52.984929 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:52.984899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:52.991082 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:52.991061    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.085554 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.085518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.085723 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.085674    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.088979 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.088956    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") DevicePath \"csi-f2a27b43c2e6b90b8d4f30f060b15f737e84b53f40b6fe582cd1bd4a72707980\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.237402 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.237339    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d9c87c9dc879eccd47f485be990b71dd6d82148023a277c5634bac4e48a8c67/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.250724 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.250696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"e2e-pvc-model-download-qcwtq\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.282973 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.282944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:24:53.426277 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.426247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"]
Jul 28 02:24:53.427913 ip-10-0-138-218 kubenswrapper[2577]: W0728 02:24:53.427882    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ed9b5b2_f75d_4fa5_8c0d_219bb6cc612b.slice/crio-db668237c51e3bf91ddcb756adfda42a26fbf0bda72de8203cd8ac732b5155aa WatchSource:0}: Error finding container db668237c51e3bf91ddcb756adfda42a26fbf0bda72de8203cd8ac732b5155aa: Status 404 returned error can't find the container with id db668237c51e3bf91ddcb756adfda42a26fbf0bda72de8203cd8ac732b5155aa
Jul 28 02:24:53.600243 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.600203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq" event={"ID":"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b","Type":"ContainerStarted","Data":"3b9ba826cc393fa6392e32bd693d74619cac41ddb9afc05ff95e2db8238c9824"}
Jul 28 02:24:53.600475 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.600251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq" event={"ID":"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b","Type":"ContainerStarted","Data":"db668237c51e3bf91ddcb756adfda42a26fbf0bda72de8203cd8ac732b5155aa"}
Jul 28 02:24:53.616046 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:24:53.615998    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq" podStartSLOduration=6.6159827490000005 podStartE2EDuration="6.615982749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:24:53.614689523 +0000 UTC m=+3544.449437070" watchObservedRunningTime="2026-07-28 02:24:53.615982749 +0000 UTC m=+3544.450730294"
Jul 28 02:25:01.646660 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:01.646620    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b" containerID="3b9ba826cc393fa6392e32bd693d74619cac41ddb9afc05ff95e2db8238c9824" exitCode=0
Jul 28 02:25:01.647254 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:01.646690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq" event={"ID":"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b","Type":"ContainerDied","Data":"3b9ba826cc393fa6392e32bd693d74619cac41ddb9afc05ff95e2db8238c9824"}
Jul 28 02:25:02.803320 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.803292    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:25:02.880716 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.880678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m8vz\" (UniqueName: \"kubernetes.io/projected/0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b-kube-api-access-6m8vz\") pod \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") "
Jul 28 02:25:02.880900 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.880812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\" (UID: \"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b\") "
Jul 28 02:25:02.882987 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.882962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b-kube-api-access-6m8vz" (OuterVolumeSpecName: "kube-api-access-6m8vz") pod "0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b" (UID: "0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b"). InnerVolumeSpecName "kube-api-access-6m8vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:25:02.883490 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.883472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc" (OuterVolumeSpecName: "model-storage") pod "0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b" (UID: "0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b"). InnerVolumeSpecName "pvc-decca439-7a9a-44ad-a2b8-7536bcf29191". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 02:25:02.982203 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.982173    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8vz\" (UniqueName: \"kubernetes.io/projected/0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b-kube-api-access-6m8vz\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 02:25:02.982334 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:02.982215    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") on node \"ip-10-0-138-218.ec2.internal\" "
Jul 28 02:25:03.093774 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:03.093752    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-decca439-7a9a-44ad-a2b8-7536bcf29191" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc") on node "ip-10-0-138-218.ec2.internal"
Jul 28 02:25:03.183480 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:03.183451    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"csi-f2a27b43c2e6b90b8d4f30f060b15f737e84b53f40b6fe582cd1bd4a72707980\""
Jul 28 02:25:03.658743 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:03.658715    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq"
Jul 28 02:25:03.658952 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:03.658715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq" event={"ID":"0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b","Type":"ContainerDied","Data":"db668237c51e3bf91ddcb756adfda42a26fbf0bda72de8203cd8ac732b5155aa"}
Jul 28 02:25:03.658952 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:03.658833    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db668237c51e3bf91ddcb756adfda42a26fbf0bda72de8203cd8ac732b5155aa"
Jul 28 02:25:24.582826 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.582786    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 02:25:24.583433 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.583412    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b" containerName="storage-initializer"
Jul 28 02:25:24.583510 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.583437    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b" containerName="storage-initializer"
Jul 28 02:25:24.583577 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.583561    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed9b5b2-f75d-4fa5-8c0d-219bb6cc612b" containerName="storage-initializer"
Jul 28 02:25:24.586643 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.586619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.590206 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.590184    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 02:25:24.590345 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.590234    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 02:25:24.590345 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.590188    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw\""
Jul 28 02:25:24.590345 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.590185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5jrnk\""
Jul 28 02:25:24.590529 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.590364    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 02:25:24.597346 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.597319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 02:25:24.684345 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.684309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.684517 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.684366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d663462d-861e-4d3e-9cc8-4fa38eda105e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.684517 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.684410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.684517 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.684488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.684631 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.684539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.684631 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.684558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztlv8\" (UniqueName: \"kubernetes.io/projected/d663462d-861e-4d3e-9cc8-4fa38eda105e-kube-api-access-ztlv8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786013 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.785982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786196 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d663462d-861e-4d3e-9cc8-4fa38eda105e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786196 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786196 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786196 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786448 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlv8\" (UniqueName: \"kubernetes.io/projected/d663462d-861e-4d3e-9cc8-4fa38eda105e-kube-api-access-ztlv8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786541 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786606 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.786606 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.786526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.788420 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.788389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d663462d-861e-4d3e-9cc8-4fa38eda105e-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.788607 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.788587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d663462d-861e-4d3e-9cc8-4fa38eda105e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:24.794383 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:24.794359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlv8\" (UniqueName: \"kubernetes.io/projected/d663462d-861e-4d3e-9cc8-4fa38eda105e-kube-api-access-ztlv8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.350455 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.350417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.355793 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.355767    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.451692 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.451657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.451894 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.451794    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.455155 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.455126    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") DevicePath \"csi-f2a27b43c2e6b90b8d4f30f060b15f737e84b53f40b6fe582cd1bd4a72707980\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.544838 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.544807    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d9c87c9dc879eccd47f485be990b71dd6d82148023a277c5634bac4e48a8c67/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.557412 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.557383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-decca439-7a9a-44ad-a2b8-7536bcf29191\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ee7dead5aceb9ddc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d663462d-861e-4d3e-9cc8-4fa38eda105e\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.798817 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.798785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 28 02:25:31.943078 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:31.940783    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 28 02:25:32.830549 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:32.830508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"d663462d-861e-4d3e-9cc8-4fa38eda105e","Type":"ContainerStarted","Data":"91c2d5ed686193244c9a131b811bca5011c1a370a307740456a339e2ff8f77ea"}
Jul 28 02:25:32.831041 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:32.830563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"d663462d-861e-4d3e-9cc8-4fa38eda105e","Type":"ContainerStarted","Data":"ee87440440f7041923a5a04757fc44ec5a4f24991f88cc1c94f78deaab5fb5e5"}
Jul 28 02:25:32.849412 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:32.849364    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=8.849349551 podStartE2EDuration="8.849349551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:25:32.84661794 +0000 UTC m=+3583.681365478" watchObservedRunningTime="2026-07-28 02:25:32.849349551 +0000 UTC m=+3583.684097130"
Jul 28 02:25:51.809067 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:51.808966    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:25:51.832835 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:25:51.832811    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:30:51.847649 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:30:51.847550    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:30:51.872933 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:30:51.872911    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:35:51.885210 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:35:51.885103    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:35:51.912396 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:35:51.912377    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:40:09.567994 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:40:09.567953    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7c3633d1_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_d663462d-861e-4d3e-9cc8-4fa38eda105e/main/0.log"
Jul 28 02:40:51.923505 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:40:51.923385    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:40:51.951842 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:40:51.951822    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:45:51.961854 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:45:51.961750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:45:52.000638 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:45:52.000605    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:50:52.003663 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:50:52.003631    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:50:52.039999 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:50:52.039974    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:55:52.040512 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:55:52.040413    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 02:55:52.081078 ip-10-0-138-218 kubenswrapper[2577]: I0728 02:55:52.081052    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 03:00:52.078931 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:00:52.078820    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 03:00:52.118028 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:00:52.118009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 03:03:20.288086 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:20.288049    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7584bddd84-xc2hz"]
Jul 28 03:03:20.288619 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:20.288243    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz" podUID="c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" containerName="manager" containerID="cri-o://edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324" gracePeriod=30
Jul 28 03:03:21.043157 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.043131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 03:03:21.207886 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.207787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-cert\") pod \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") "
Jul 28 03:03:21.207886 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.207847    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhj2d\" (UniqueName: \"kubernetes.io/projected/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-kube-api-access-hhj2d\") pod \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\" (UID: \"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85\") "
Jul 28 03:03:21.209842 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.209817    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-cert" (OuterVolumeSpecName: "cert") pod "c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" (UID: "c7e5a9ad-6752-4e1e-bdcc-96757a77bd85"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:03:21.210010 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.209989    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-kube-api-access-hhj2d" (OuterVolumeSpecName: "kube-api-access-hhj2d") pod "c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" (UID: "c7e5a9ad-6752-4e1e-bdcc-96757a77bd85"). InnerVolumeSpecName "kube-api-access-hhj2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:03:21.309537 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.309509    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhj2d\" (UniqueName: \"kubernetes.io/projected/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-kube-api-access-hhj2d\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 03:03:21.309537 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.309532    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85-cert\") on node \"ip-10-0-138-218.ec2.internal\" DevicePath \"\""
Jul 28 03:03:21.877337 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.877306    2577 generic.go:356] "Generic (PLEG): container finished" podID="c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" containerID="edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324" exitCode=0
Jul 28 03:03:21.877522 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.877351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz"
Jul 28 03:03:21.877522 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.877348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz" event={"ID":"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85","Type":"ContainerDied","Data":"edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324"}
Jul 28 03:03:21.877522 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.877461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7584bddd84-xc2hz" event={"ID":"c7e5a9ad-6752-4e1e-bdcc-96757a77bd85","Type":"ContainerDied","Data":"0e04adfb54aa7eee83d751bb1f0b139d0123159683f25307fd1379e6a3367b87"}
Jul 28 03:03:21.877522 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.877484    2577 scope.go:117] "RemoveContainer" containerID="edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324"
Jul 28 03:03:21.893746 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.893727    2577 scope.go:117] "RemoveContainer" containerID="edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324"
Jul 28 03:03:21.894024 ip-10-0-138-218 kubenswrapper[2577]: E0728 03:03:21.894000    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324\": container with ID starting with edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324 not found: ID does not exist" containerID="edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324"
Jul 28 03:03:21.894128 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.894030    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324"} err="failed to get container status \"edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324\": rpc error: code = NotFound desc = could not find container \"edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324\": container with ID starting with edb6ba37b30fa55fadd2eeed01aeafabba308ecec87b05ae540b4150e0bc8324 not found: ID does not exist"
Jul 28 03:03:21.896111 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.896090    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7584bddd84-xc2hz"]
Jul 28 03:03:21.898363 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:21.898344    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-7584bddd84-xc2hz"]
Jul 28 03:03:23.697662 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:03:23.697628    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" path="/var/lib/kubelet/pods/c7e5a9ad-6752-4e1e-bdcc-96757a77bd85/volumes"
Jul 28 03:05:52.115501 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:05:52.115403    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 03:05:52.159290 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:05:52.159268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 03:08:29.699288 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:29.699257    2577 ???:1] "http: TLS handshake error from 10.0.139.81:54720: EOF"
Jul 28 03:08:29.966590 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:29.966517    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:08:30.908317 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:30.908285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:08:31.848643 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:31.848612    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:08:32.802366 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:32.802332    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:08:33.669639 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:33.669610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:33.686322 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:33.686296    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:33.705416 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:33.705390    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:33.902063 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:33.902036    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:34.788426 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:34.788395    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:34.903216 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:34.903184    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:34.934000 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:34.933977    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:35.147684 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:35.147598    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:36.156364 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:36.156329    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:36.180311 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:36.180280    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:36.196971 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:36.196949    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:36.392675 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:36.392644    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:37.234497 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:37.234462    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:37.254573 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:37.254546    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:37.273158 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:37.273137    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:37.471242 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:37.471201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:38.303809 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:38.303773    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:38.321667 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:38.321628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:38.336071 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:38.336051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:38.530600 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:38.530564    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:39.392696 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:39.392660    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:39.412979 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:39.412957    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:39.430068 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:39.430051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:39.636378 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:39.636341    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:40.494380 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:40.494347    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:40.516333 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:40.516307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:40.530334 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:40.530313    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:40.725050 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:40.725015    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:41.556956 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:41.556923    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:41.577388 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:41.577352    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:41.590519 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:41.590493    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:41.786351 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:41.786314    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:42.612148 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:42.612116    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:42.632563 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:42.632536    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:42.650614 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:42.650591    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:42.852690 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:42.852656    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:43.717064 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:43.717035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:43.738905 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:43.738856    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:43.754740 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:43.754716    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:43.957664 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:43.957629    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:44.826376 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:44.826342    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:44.847672 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:44.847643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:44.862712 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:44.862688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:45.070388 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:45.070351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:45.918698 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:45.918663    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:45.936924 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:45.936901    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:45.953154 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:45.953131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:46.147291 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:46.147255    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:47.037146 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:47.037110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:47.055749 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:47.055718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:47.070236 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:47.070216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:47.267573 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:47.267544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:48.137532 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:48.137496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:48.156465 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:48.156436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:48.173631 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:48.173608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:48.381947 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:48.381914    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:49.226957 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:49.226921    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:49.246039 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:49.245995    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:49.258526 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:49.258500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:49.464575 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:49.464544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:50.335569 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:50.335542    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:50.352801 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:50.352775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:50.367925 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:50.367900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:50.569570 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:50.569528    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:51.415481 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:51.415442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:51.436597 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:51.436575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:51.451259 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:51.451234    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:51.653186 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:51.653147    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:52.499813 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:52.499783    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:52.519850 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:52.519819    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:52.536156 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:52.536135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:52.732219 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:52.732186    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:53.572515 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:53.572482    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:53.589297 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:53.589271    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:53.603268 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:53.603229    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:53.793782 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:53.793743    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:54.640440 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:54.640406    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:54.659374 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:54.659339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:54.674852 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:54.674818    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:54.867745 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:54.867705    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:55.716577 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:55.716530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:55.735833 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:55.735806    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:55.750368 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:55.750345    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:55.946220 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:55.946191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:56.783515 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:56.783476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:56.801328 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:56.801301    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:56.814644 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:56.814618    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:57.014137 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:57.014101    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:57.857185 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:57.857149    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:57.876307 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:57.876277    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:57.890750 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:57.890725    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:58.080630 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:58.080581    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:08:58.945882 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:58.945824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:08:58.975931 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:58.975897    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:08:58.996368 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:58.996339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:08:59.192620 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:08:59.192590    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:09:00.066529 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:00.066492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:09:00.086196 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:00.086169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:09:00.099920 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:00.099895    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:09:00.300618 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:00.300588    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:09:04.974224 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:04.974194    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pqlq7_907860fa-083a-47ae-a4ad-dc7776e93632/global-pull-secret-syncer/0.log"
Jul 28 03:09:05.158047 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:05.158012    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wthds_ce65bb36-da1a-4be4-84cd-b01bc0fabece/konnectivity-agent/0.log"
Jul 28 03:09:05.182446 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:05.182415    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-218.ec2.internal_4c0e38832605850b3fb0e1ff07703dec/haproxy/0.log"
Jul 28 03:09:10.186689 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:10.186652    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-4n9wp_e8511451-bfc7-4681-ad2c-4869d499a685/authorino/0.log"
Jul 28 03:09:10.212003 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:10.211975    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-vrqn9_94f786d2-667e-45a7-8c28-87cc26a49e41/manager/0.log"
Jul 28 03:09:10.238492 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:10.238467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-5k7td_2e1ccc8a-070d-43e4-98ff-745e0738c01b/manager/0.log"
Jul 28 03:09:10.466505 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:10.463249    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-sfbvm_2286f348-7525-45b4-9e9c-05817b697241/manager/0.log"
Jul 28 03:09:11.823004 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.822935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/alertmanager/0.log"
Jul 28 03:09:11.854398 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.854371    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/config-reloader/0.log"
Jul 28 03:09:11.881674 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.881648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/kube-rbac-proxy-web/0.log"
Jul 28 03:09:11.900174 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.900152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/kube-rbac-proxy/0.log"
Jul 28 03:09:11.925936 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.925893    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/kube-rbac-proxy-metric/0.log"
Jul 28 03:09:11.950545 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.950520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/prom-label-proxy/0.log"
Jul 28 03:09:11.981583 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:11.981558    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d0ba98e6-e092-4907-b35a-9c427cffaaaa/init-config-reloader/0.log"
Jul 28 03:09:12.034627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.034593    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-9vgj6_185e9e72-f552-4f09-aebc-51714ec10f67/cluster-monitoring-operator/0.log"
Jul 28 03:09:12.145405 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.145323    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c5b67497c-gr5jc_8f263054-4032-44e7-9566-15acc9322453/metrics-server/0.log"
Jul 28 03:09:12.174914 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.174883    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-5bx4k_9fc2a0c8-220b-4a5f-8ca9-c8a9382e9d84/monitoring-plugin/0.log"
Jul 28 03:09:12.294851 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.294822    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ks6hc_ec575a4e-f6d7-495f-9b5b-cf38ac5107e4/node-exporter/0.log"
Jul 28 03:09:12.316929 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.316905    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ks6hc_ec575a4e-f6d7-495f-9b5b-cf38ac5107e4/kube-rbac-proxy/0.log"
Jul 28 03:09:12.349036 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.349005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ks6hc_ec575a4e-f6d7-495f-9b5b-cf38ac5107e4/init-textfile/0.log"
Jul 28 03:09:12.485239 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.485210    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-47wrj_cee46962-4618-4584-8574-622956fbf5e0/kube-rbac-proxy-main/0.log"
Jul 28 03:09:12.509730 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.509696    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-47wrj_cee46962-4618-4584-8574-622956fbf5e0/kube-rbac-proxy-self/0.log"
Jul 28 03:09:12.535705 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.535679    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-47wrj_cee46962-4618-4584-8574-622956fbf5e0/openshift-state-metrics/0.log"
Jul 28 03:09:12.839747 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.839718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58476696fd-pthfz_eb38993c-646f-47cc-a3ac-9759b09df6f6/telemeter-client/0.log"
Jul 28 03:09:12.863201 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.863171    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58476696fd-pthfz_eb38993c-646f-47cc-a3ac-9759b09df6f6/reload/0.log"
Jul 28 03:09:12.892139 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:12.892110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58476696fd-pthfz_eb38993c-646f-47cc-a3ac-9759b09df6f6/kube-rbac-proxy/0.log"
Jul 28 03:09:14.799641 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:14.799613    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/1.log"
Jul 28 03:09:14.804906 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:14.804887    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ms892_d0b4a55f-79d1-43d3-92a6-aa7c5f3606fa/console-operator/2.log"
Jul 28 03:09:15.059082 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.059001    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"]
Jul 28 03:09:15.059400 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.059388    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" containerName="manager"
Jul 28 03:09:15.059443 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.059402    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" containerName="manager"
Jul 28 03:09:15.059489 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.059480    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e5a9ad-6752-4e1e-bdcc-96757a77bd85" containerName="manager"
Jul 28 03:09:15.062663 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.062644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.065025 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.065001    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q4dh\"/\"kube-root-ca.crt\""
Jul 28 03:09:15.065231 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.065211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q4dh\"/\"openshift-service-ca.crt\""
Jul 28 03:09:15.065979 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.065963    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6q4dh\"/\"default-dockercfg-fvqfh\""
Jul 28 03:09:15.069515 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.069486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"]
Jul 28 03:09:15.166041 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.166007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsh6r\" (UniqueName: \"kubernetes.io/projected/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-kube-api-access-tsh6r\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.166229 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.166071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-proc\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.166229 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.166158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-podres\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.166229 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.166184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-sys\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.166411 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.166228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-lib-modules\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267448 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-podres\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-sys\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-lib-modules\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsh6r\" (UniqueName: \"kubernetes.io/projected/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-kube-api-access-tsh6r\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-proc\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-sys\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267627 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-podres\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267928 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-proc\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.267928 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.267718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-lib-modules\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.275666 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.275636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsh6r\" (UniqueName: \"kubernetes.io/projected/1ea72fbf-5123-4b5a-96d1-c9d0fc20d979-kube-api-access-tsh6r\") pod \"perf-node-gather-daemonset-9w7rj\" (UID: \"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.373928 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.373806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.514044 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.514010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"]
Jul 28 03:09:15.515709 ip-10-0-138-218 kubenswrapper[2577]: W0728 03:09:15.515681    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1ea72fbf_5123_4b5a_96d1_c9d0fc20d979.slice/crio-102b0c99db22c730f04cab1075f5d075f974483ad67f452c9c1beab27a3c9e12 WatchSource:0}: Error finding container 102b0c99db22c730f04cab1075f5d075f974483ad67f452c9c1beab27a3c9e12: Status 404 returned error can't find the container with id 102b0c99db22c730f04cab1075f5d075f974483ad67f452c9c1beab27a3c9e12
Jul 28 03:09:15.875031 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.874997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj" event={"ID":"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979","Type":"ContainerStarted","Data":"9db77d672703ead6d464947e3dbdf7b9bc01054cb3df9e8298b4f0e2a93fc7a9"}
Jul 28 03:09:15.875474 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.875056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:15.875474 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.875077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj" event={"ID":"1ea72fbf-5123-4b5a-96d1-c9d0fc20d979","Type":"ContainerStarted","Data":"102b0c99db22c730f04cab1075f5d075f974483ad67f452c9c1beab27a3c9e12"}
Jul 28 03:09:15.892717 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:15.892672    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj" podStartSLOduration=0.892656301 podStartE2EDuration="892.656301ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 03:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:09:15.889765376 +0000 UTC m=+6206.724512922" watchObservedRunningTime="2026-07-28 03:09:15.892656301 +0000 UTC m=+6206.727403846"
Jul 28 03:09:16.929000 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:16.928972    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpzzq_b6df3021-0059-41bc-be13-3067ea24b062/dns/0.log"
Jul 28 03:09:16.954347 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:16.954319    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jpzzq_b6df3021-0059-41bc-be13-3067ea24b062/kube-rbac-proxy/0.log"
Jul 28 03:09:17.046107 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:17.046072    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-74w4r_6e7dba7a-71b6-4c1a-a3fd-ad2e7b92ba67/dns-node-resolver/0.log"
Jul 28 03:09:17.717827 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:17.717795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-659f8869dc-t2rzb_fe286845-7f5c-4f55-a96a-21cc7efc794f/registry/0.log"
Jul 28 03:09:17.768520 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:17.768487    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2thtx_f0e34270-2b92-4098-93e6-028f9fe3f1ee/node-ca/0.log"
Jul 28 03:09:18.926051 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:18.926026    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:09:19.503645 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:19.503618    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7v6r_764abbf1-6c23-4e88-adca-d80b035509ce/serve-healthcheck-canary/0.log"
Jul 28 03:09:20.010521 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:20.010491    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-6k54l_6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3/insights-operator/0.log"
Jul 28 03:09:20.013087 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:20.013069    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-6k54l_6b7b9cf5-039e-4b0d-a0a8-e3b711f72cc3/insights-operator/1.log"
Jul 28 03:09:20.108513 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:20.108485    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h5pr8_9f5c33a5-68fd-46cc-bd88-abb9aff18dfd/kube-rbac-proxy/0.log"
Jul 28 03:09:20.131131 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:20.131107    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h5pr8_9f5c33a5-68fd-46cc-bd88-abb9aff18dfd/exporter/0.log"
Jul 28 03:09:20.153137 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:20.153108    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h5pr8_9f5c33a5-68fd-46cc-bd88-abb9aff18dfd/extractor/0.log"
Jul 28 03:09:21.892536 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:21.892511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-9w7rj"
Jul 28 03:09:23.844577 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:23.844552    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-rs8w6_fba12f0b-a968-411a-8246-a85542a7f2f2/manager/0.log"
Jul 28 03:09:23.940060 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:23.940034    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dj9c7_85edf2f2-b337-454c-b394-f364b2dfa2de/server/0.log"
Jul 28 03:09:24.629360 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:24.629330    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-b5sgr_5d0541bf-1ff2-40e7-b7f8-0e1946f6f6b4/manager/0.log"
Jul 28 03:09:24.666103 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:24.666076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-twkpm_abc7c1b5-7021-42d4-a3fa-49f39a005ead/s3-init/0.log"
Jul 28 03:09:24.686035 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:24.686009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-twkpm_abc7c1b5-7021-42d4-a3fa-49f39a005ead/download-hf-model/0.log"
Jul 28 03:09:24.950686 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:24.950651    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:09:26.281113 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:26.281055    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4r9gw_bc3aed44-1564-4f05-a8a8-2f3b536464a2/discovery/0.log"
Jul 28 03:09:31.607903 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.607851    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/kube-multus-additional-cni-plugins/0.log"
Jul 28 03:09:31.642034 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.642005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/egress-router-binary-copy/0.log"
Jul 28 03:09:31.680116 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.680084    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/cni-plugins/0.log"
Jul 28 03:09:31.724160 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.723999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/bond-cni-plugin/0.log"
Jul 28 03:09:31.772033 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.772006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/routeoverride-cni/0.log"
Jul 28 03:09:31.809041 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.809017    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/whereabouts-cni-bincopy/0.log"
Jul 28 03:09:31.845814 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.845790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sf8nb_2d0bd5ff-4e9c-4b42-a919-ce3ade437a55/whereabouts-cni/0.log"
Jul 28 03:09:31.898453 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:31.898387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-chg9p_695e801e-7959-454a-9698-e492ff548233/kube-multus/0.log"
Jul 28 03:09:32.168269 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:32.168196    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qbb85_fb8c2454-c3c5-4499-a2d8-1a5cfe353eba/network-metrics-daemon/0.log"
Jul 28 03:09:32.203193 ip-10-0-138-218 kubenswrapper[2577]: I0728 03:09:32.203153    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qbb85_fb8c2454-c3c5-4499-a2d8-1a5cfe353eba/kube-rbac-proxy/0.log"