Apr 24 23:54:00.270886 ip-10-0-134-82 systemd[1]: Starting Kubernetes Kubelet...
Apr 24 23:54:00.639454 ip-10-0-134-82 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 24 23:54:00.639454 ip-10-0-134-82 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 24 23:54:00.639454 ip-10-0-134-82 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 24 23:54:00.639454 ip-10-0-134-82 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 24 23:54:00.639454 ip-10-0-134-82 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 24 23:54:00.640456 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.640229    2576 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 24 23:54:00.642674 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642655    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 23:54:00.642674 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642673    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 23:54:00.642674 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642677    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642680    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642683    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642689    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642693    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642696    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642700    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642702    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642705    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642708    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642710    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642721    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642724    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642727    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642731    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642734    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642737    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642739    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642742    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 23:54:00.642766 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642744    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642747    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642750    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642753    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642756    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642759    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642761    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642764    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642766    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642769    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642772    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642775    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642777    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642780    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642783    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642786    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642788    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642792    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642794    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642797    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 23:54:00.643225 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642800    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642802    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642805    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642807    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642810    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642812    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642814    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642817    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642819    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642822    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642824    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642827    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642829    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642832    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642835    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642838    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642841    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642844    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642846    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642849    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 23:54:00.643717 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642851    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642854    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642856    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642858    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642861    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642864    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642867    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642869    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642872    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642875    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642877    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642880    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642882    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642884    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642887    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642890    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642892    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642895    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642897    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 23:54:00.644196 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642900    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642903    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642905    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642908    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642910    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.642913    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643303    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643308    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643312    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643315    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643318    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643321    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643323    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643326    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643329    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643331    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643333    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643336    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643339    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643342    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 23:54:00.644663 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643344    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643347    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643349    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643352    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643354    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643357    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643359    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643362    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643364    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643367    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643369    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643371    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643374    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643376    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643379    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643383    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643387    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643390    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643393    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 23:54:00.645148 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643397    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643400    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643402    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643405    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643408    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643410    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643413    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643415    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643418    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643420    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643423    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643425    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643428    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643430    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643433    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643435    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643437    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643440    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643442    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643445    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 23:54:00.645632 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643447    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643450    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643453    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643455    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643458    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643460    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643462    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643465    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643468    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643470    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643472    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643475    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643478    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643481    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643484    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643486    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643489    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643491    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643494    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643496    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 23:54:00.646119 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643499    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643501    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643504    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643506    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643508    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643512    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643515    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643517    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643521    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643523    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643526    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643530    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.643533    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644152    2576 flags.go:64] FLAG: --address="0.0.0.0"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644167    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644174    2576 flags.go:64] FLAG: --anonymous-auth="true"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644179    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644183    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644186    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644191    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644196    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 24 23:54:00.646650 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644200    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644203    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644207    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644210    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644214    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644217    2576 flags.go:64] FLAG: --cgroup-root=""
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644220    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644223    2576 flags.go:64] FLAG: --client-ca-file=""
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644225    2576 flags.go:64] FLAG: --cloud-config=""
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644228    2576 flags.go:64] FLAG: --cloud-provider="external"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644231    2576 flags.go:64] FLAG: --cluster-dns="[]"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644236    2576 flags.go:64] FLAG: --cluster-domain=""
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644238    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644242    2576 flags.go:64] FLAG: --config-dir=""
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644245    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644248    2576 flags.go:64] FLAG: --container-log-max-files="5"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644252    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644256    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644259    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644263    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644266    2576 flags.go:64] FLAG: --contention-profiling="false"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644269    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644272    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644275    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644278    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 24 23:54:00.647163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644282    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644285    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644288    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644291    2576 flags.go:64] FLAG: --enable-load-reader="false"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644294    2576 flags.go:64] FLAG: --enable-server="true"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644297    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644302    2576 flags.go:64] FLAG: --event-burst="100"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644305    2576 flags.go:64] FLAG: --event-qps="50"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644308    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644311    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644314    2576 flags.go:64] FLAG: --eviction-hard=""
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644318    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644321    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644324    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644327    2576 flags.go:64] FLAG: --eviction-soft=""
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644330    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644333    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644336    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644339    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644342    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644345    2576 flags.go:64] FLAG: --fail-swap-on="true"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644348    2576 flags.go:64] FLAG: --feature-gates=""
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644351    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644354    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644358    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 24 23:54:00.647812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644361    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644364    2576 flags.go:64] FLAG: --healthz-port="10248"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644367    2576 flags.go:64] FLAG: --help="false"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644370    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-82.ec2.internal"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644373    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644376    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644379    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644382    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644385    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644388    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644395    2576 flags.go:64] FLAG: --image-service-endpoint=""
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644398    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644401    2576 flags.go:64] FLAG: --kube-api-burst="100"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644404    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644407    2576 flags.go:64] FLAG: --kube-api-qps="50"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644409    2576 flags.go:64] FLAG: --kube-reserved=""
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644412    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644415    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644418    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644421    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644424    2576 flags.go:64] FLAG: --lock-file=""
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644427    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644430    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644433    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 24 23:54:00.648507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644439    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644442    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644445    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644447    2576 flags.go:64] FLAG: --logging-format="text"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644450    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644454    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644457    2576 flags.go:64] FLAG: --manifest-url=""
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644459    2576 flags.go:64] FLAG: --manifest-url-header=""
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644464    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644468    2576 flags.go:64] FLAG: --max-open-files="1000000"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644473    2576 flags.go:64] FLAG: --max-pods="110"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644476    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644478    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644481    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644484    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644487    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644490    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644493    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644502    2576 flags.go:64] FLAG: --node-status-max-images="50"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644505    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644508    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644511    2576 flags.go:64] FLAG: --pod-cidr=""
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644514    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715"
Apr 24 23:54:00.649119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644520    2576 flags.go:64] FLAG: --pod-manifest-path=""
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644523    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644526    2576 flags.go:64] FLAG: --pods-per-core="0"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644529    2576 flags.go:64] FLAG: --port="10250"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644532    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644535    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5d09d5f96de2445"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644538    2576 flags.go:64] FLAG: --qos-reserved=""
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644541    2576 flags.go:64] FLAG: --read-only-port="10255"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644544    2576 flags.go:64] FLAG: --register-node="true"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644547    2576 flags.go:64] FLAG: --register-schedulable="true"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644550    2576 flags.go:64] FLAG: --register-with-taints=""
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644553    2576 flags.go:64] FLAG: --registry-burst="10"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644556    2576 flags.go:64] FLAG: --registry-qps="5"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644559    2576 flags.go:64] FLAG: --reserved-cpus=""
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644562    2576 flags.go:64] FLAG: --reserved-memory=""
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644565    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644568    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644572    2576 flags.go:64] FLAG: --rotate-certificates="false"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644575    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644578    2576 flags.go:64] FLAG: --runonce="false"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644581    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644584    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644587    2576 flags.go:64] FLAG: --seccomp-default="false"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644590    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644593    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644615    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 24 23:54:00.649722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644620    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644625    2576 flags.go:64] FLAG: --storage-driver-password="root"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644631    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644633    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644636    2576 flags.go:64] FLAG: --storage-driver-user="root"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644639    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644642    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644645    2576 flags.go:64] FLAG: --system-cgroups=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644648    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644654    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644657    2576 flags.go:64] FLAG: --tls-cert-file=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644660    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644664    2576 flags.go:64] FLAG: --tls-min-version=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644667    2576 flags.go:64] FLAG: --tls-private-key-file=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644670    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644673    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644676    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644679    2576 flags.go:64] FLAG: --v="2"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644683    2576 flags.go:64] FLAG: --version="false"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644687    2576 flags.go:64] FLAG: --vmodule=""
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644691    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.644694    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644797    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644800    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644804    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 23:54:00.650379 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644807    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644810    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644813    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644816    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644820    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644823    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644825    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644828    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644831    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644834    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644836    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644839    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644841    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644844    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644846    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644849    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644852    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644854    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644857    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644859    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 23:54:00.651054 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644862    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644864    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644867    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644870    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644872    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644874    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644877    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644879    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644882    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644886    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644889    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644892    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644895    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644898    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644901    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644904    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644908    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644911    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644913    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 23:54:00.651571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644916    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644919    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644922    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644925    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644927    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644931    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644934    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644937    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644940    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644942    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644945    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644948    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644950    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644953    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644955    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644958    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644960    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644962    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644965    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644967    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 23:54:00.652065 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644970    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644972    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644975    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644977    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644980    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644983    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644985    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644988    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644991    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644995    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644997    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.644999    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645002    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645004    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645007    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645010    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645012    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645015    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645017    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645020    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 23:54:00.652561 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645022    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645024    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645027    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.645030    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.645634    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.652438    2576 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.652456    2576 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652522    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652528    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652530    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652533    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652536    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652539    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652542    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652545    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 23:54:00.653088 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652547    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652550    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652552    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652554    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652557    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652560    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652563    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652565    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652568    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652571    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652573    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652576    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652579    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652581    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652584    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652586    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652589    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652591    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652594    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652617    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 23:54:00.653473 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652621    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652624    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652627    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652637    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652640    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652643    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652645    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652648    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652650    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652653    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652656    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652658    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652661    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652664    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652666    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652669    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652671    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652673    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652676    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652678    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 23:54:00.653995 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652681    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652684    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652688    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652691    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652694    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652697    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652705    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652707    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652710    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652713    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652715    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652718    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652720    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652723    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652726    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652729    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652737    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652740    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652742    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 23:54:00.654533 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652745    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652747    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652750    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652752    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652754    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652757    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652759    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652762    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652764    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652766    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652769    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652771    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652774    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652776    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652779    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652781    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652783    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652786    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 23:54:00.655025 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652788    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.652793    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652923    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652928    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652931    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652933    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652936    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652938    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652941    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652944    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652947    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652949    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652958    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652960    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652963    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 23:54:00.655458 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652965    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652968    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652970    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652973    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652976    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652978    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652981    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652983    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652987    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652990    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652994    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.652997    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653000    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653003    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653006    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653008    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653011    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653014    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653016    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 23:54:00.655851 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653018    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653021    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653023    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653026    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653028    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653031    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653033    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653036    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653038    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653041    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653043    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653052    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653055    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653058    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653060    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653063    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653065    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653067    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653070    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653072    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 23:54:00.656323 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653074    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653077    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653079    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653082    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653084    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653087    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653089    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653092    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653094    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653097    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653100    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653102    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653105    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653108    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653110    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653112    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653115    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653117    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653119    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653122    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 23:54:00.656964 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653124    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653127    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653129    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653137    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653145    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653148    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653150    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653152    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653155    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653157    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653160    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653162    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653164    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:00.653167    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.653172    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 24 23:54:00.657462 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.653325    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 24 23:54:00.657889 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.656213    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 24 23:54:00.657889 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.657256    2576 server.go:1019] "Starting client certificate rotation"
Apr 24 23:54:00.657889 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.657369    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 24 23:54:00.658048 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.658037    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 24 23:54:00.680105 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.680075    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 24 23:54:00.682332 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.682305    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 24 23:54:00.693679 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.693655    2576 log.go:25] "Validated CRI v1 runtime API"
Apr 24 23:54:00.698561 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.698537    2576 log.go:25] "Validated CRI v1 image API"
Apr 24 23:54:00.699772 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.699751    2576 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 24 23:54:00.703165 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.703135    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d95d3ca3-89f1-490f-b589-a200993bc57a:/dev/nvme0n1p3 e044c5f4-44b9-4d79-ab63-b459203dfce2:/dev/nvme0n1p4]
Apr 24 23:54:00.703257 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.703162    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 24 23:54:00.709202 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.709178    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 24 23:54:00.709347 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.709074    2576 manager.go:217] Machine: {Timestamp:2026-04-24 23:54:00.707374201 +0000 UTC m=+0.332413868 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099334 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b584c9ac4d82c12edc2ea58468e00 SystemUUID:ec2b584c-9ac4-d82c-12ed-c2ea58468e00 BootID:fb920fed-26f6-48f3-8546-f6faf6a6ac5f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fd:99:53:e2:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fd:99:53:e2:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:b7:bc:73:8c:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 24 23:54:00.709347 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.709325    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 24 23:54:00.709533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.709510    2576 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 24 23:54:00.711426 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.711383    2576 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 24 23:54:00.711627 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.711428    2576 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 24 23:54:00.711680 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.711639    2576 topology_manager.go:138] "Creating topology manager with none policy"
Apr 24 23:54:00.711680 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.711648    2576 container_manager_linux.go:306] "Creating device plugin manager"
Apr 24 23:54:00.711680 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.711662    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 24 23:54:00.712089 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.712079    2576 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 24 23:54:00.712902 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.712892    2576 state_mem.go:36] "Initialized new in-memory state store"
Apr 24 23:54:00.713039 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.713030    2576 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 24 23:54:00.715557 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.715545    2576 kubelet.go:491] "Attempting to sync node with API server"
Apr 24 23:54:00.715593 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.715563    2576 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 24 23:54:00.715593 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.715582    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 24 23:54:00.715593 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.715591    2576 kubelet.go:397] "Adding apiserver pod source"
Apr 24 23:54:00.715727 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.715618    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 24 23:54:00.716567 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.716556    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 24 23:54:00.716622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.716574    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 24 23:54:00.719407 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.719384    2576 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 24 23:54:00.720740 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.720723    2576 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 24 23:54:00.722115 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722103    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722120    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722126    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722132    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722138    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722143    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722149    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722154    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722161    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 24 23:54:00.722164 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722167    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 24 23:54:00.722409 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722176    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 24 23:54:00.722409 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722184    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 24 23:54:00.722866 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722852    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 24 23:54:00.722903 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.722871    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 24 23:54:00.726702 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.726686    2576 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 24 23:54:00.726793 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.726733    2576 server.go:1295] "Started kubelet"
Apr 24 23:54:00.726849 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.726797    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 24 23:54:00.726951 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.726881    2576 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 24 23:54:00.727002 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.726980    2576 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 24 23:54:00.727310 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.727294    2576 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Apr 24 23:54:00.727401 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.727381    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 24 23:54:00.727401 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.727390    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 24 23:54:00.727680 ip-10-0-134-82 systemd[1]: Started Kubernetes Kubelet.
Apr 24 23:54:00.727947 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.727929    2576 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 24 23:54:00.729481 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.729464    2576 server.go:317] "Adding debug handlers to kubelet server"
Apr 24 23:54:00.734352 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.734331    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 24 23:54:00.734736 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.734721    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 24 23:54:00.734934 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.734909    2576 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Apr 24 23:54:00.735192 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735177    2576 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 24 23:54:00.735257 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735197    2576 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 24 23:54:00.735257 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735242    2576 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 24 23:54:00.735353 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735296    2576 reconstruct.go:97] "Volume reconstruction finished"
Apr 24 23:54:00.735353 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735304    2576 reconciler.go:26] "Reconciler: start to sync state"
Apr 24 23:54:00.735534 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.735507    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:00.735534 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735514    2576 factory.go:55] Registering systemd factory
Apr 24 23:54:00.735680 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735587    2576 factory.go:223] Registration of the systemd container factory successfully
Apr 24 23:54:00.735831 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735817    2576 factory.go:153] Registering CRI-O factory
Apr 24 23:54:00.735890 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735834    2576 factory.go:223] Registration of the crio container factory successfully
Apr 24 23:54:00.735890 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735873    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 24 23:54:00.735945 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735892    2576 factory.go:103] Registering Raw factory
Apr 24 23:54:00.735945 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.735902    2576 manager.go:1196] Started watching for new ooms in manager
Apr 24 23:54:00.736260 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.736249    2576 manager.go:319] Starting recovery of all containers
Apr 24 23:54:00.738325 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.738289    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Apr 24 23:54:00.738428 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.738409    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Apr 24 23:54:00.741619 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.737861    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-82.ec2.internal.18a97026fccc825b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-82.ec2.internal,UID:ip-10-0-134-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-82.ec2.internal,},FirstTimestamp:2026-04-24 23:54:00.726700635 +0000 UTC m=+0.351740309,LastTimestamp:2026-04-24 23:54:00.726700635 +0000 UTC m=+0.351740309,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-82.ec2.internal,}"
Apr 24 23:54:00.745926 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.745816    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kslqs"
Apr 24 23:54:00.750021 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.750002    2576 manager.go:324] Recovery completed
Apr 24 23:54:00.754206 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.754193    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 23:54:00.757163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757137    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 23:54:00.757237 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757168    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 23:54:00.757237 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757179    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientPID"
Apr 24 23:54:00.757319 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757236    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kslqs"
Apr 24 23:54:00.757742 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757730    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 24 23:54:00.757742 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757741    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 24 23:54:00.757830 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.757754    2576 state_mem.go:36] "Initialized new in-memory state store"
Apr 24 23:54:00.759109 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.759043    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-82.ec2.internal.18a97026fe9d40c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-82.ec2.internal,UID:ip-10-0-134-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-82.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-82.ec2.internal,},FirstTimestamp:2026-04-24 23:54:00.757158089 +0000 UTC m=+0.382197756,LastTimestamp:2026-04-24 23:54:00.757158089 +0000 UTC m=+0.382197756,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-82.ec2.internal,}"
Apr 24 23:54:00.761200 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.761182    2576 policy_none.go:49] "None policy: Start"
Apr 24 23:54:00.761200 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.761200    2576 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 24 23:54:00.761343 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.761212    2576 state_mem.go:35] "Initializing new in-memory state store"
Apr 24 23:54:00.808382 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808359    2576 manager.go:341] "Starting Device Plugin manager"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.808408    2576 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808420    2576 server.go:85] "Starting device plugin registration server"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808728    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808748    2576 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808864    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808948    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.808957    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.809741    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 24 23:54:00.816497 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.809775    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:00.867380 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.867337    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 24 23:54:00.868764 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.868734    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 24 23:54:00.868893 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.868770    2576 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 24 23:54:00.868893 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.868799    2576 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 24 23:54:00.868893 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.868808    2576 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 24 23:54:00.869039 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.868905    2576 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 24 23:54:00.870796 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.870772    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 23:54:00.909938 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.909848    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 23:54:00.910838 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.910820    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 23:54:00.910952 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.910854    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 23:54:00.910952 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.910868    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientPID"
Apr 24 23:54:00.910952 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.910900    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-82.ec2.internal"
Apr 24 23:54:00.918853 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.918837    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-82.ec2.internal"
Apr 24 23:54:00.918916 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.918860    2576 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-82.ec2.internal\": node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:00.933807 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.933773    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:00.969772 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.969745    2576 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"]
Apr 24 23:54:00.969863 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.969818    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 23:54:00.970709 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.970692    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 23:54:00.970799 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.970725    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 23:54:00.970799 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.970739    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientPID"
Apr 24 23:54:00.973063 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973051    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 23:54:00.973213 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973200    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:00.973249 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973228    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 23:54:00.973740 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973725    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 23:54:00.973821 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973728    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 23:54:00.973821 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973780    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 23:54:00.973821 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973798    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientPID"
Apr 24 23:54:00.973821 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973750    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 23:54:00.973991 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.973826    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientPID"
Apr 24 23:54:00.975896 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.975878    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:00.976003 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.975903    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 23:54:00.976528 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.976515    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 23:54:00.976587 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.976539    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 23:54:00.976587 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:00.976553    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeHasSufficientPID"
Apr 24 23:54:00.999379 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:00.999357    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-82.ec2.internal\" not found" node="ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.003694 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.003675    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-82.ec2.internal\" not found" node="ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.034842 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.034807    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.135059 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.135021    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.136241 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.136221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52bd46895be952cff06d137a21c827f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal\" (UID: \"52bd46895be952cff06d137a21c827f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.136335 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.136273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52bd46895be952cff06d137a21c827f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal\" (UID: \"52bd46895be952cff06d137a21c827f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.136335 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.136303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20602652b465b1663ffa2d71aa406f85-config\") pod \"kube-apiserver-proxy-ip-10-0-134-82.ec2.internal\" (UID: \"20602652b465b1663ffa2d71aa406f85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.235211 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.235144    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.237356 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.237341    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52bd46895be952cff06d137a21c827f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal\" (UID: \"52bd46895be952cff06d137a21c827f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.237412 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.237364    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52bd46895be952cff06d137a21c827f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal\" (UID: \"52bd46895be952cff06d137a21c827f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.237412 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.237383    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20602652b465b1663ffa2d71aa406f85-config\") pod \"kube-apiserver-proxy-ip-10-0-134-82.ec2.internal\" (UID: \"20602652b465b1663ffa2d71aa406f85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.237412 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.237408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20602652b465b1663ffa2d71aa406f85-config\") pod \"kube-apiserver-proxy-ip-10-0-134-82.ec2.internal\" (UID: \"20602652b465b1663ffa2d71aa406f85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.237534 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.237433    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52bd46895be952cff06d137a21c827f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal\" (UID: \"52bd46895be952cff06d137a21c827f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.237534 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.237454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52bd46895be952cff06d137a21c827f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal\" (UID: \"52bd46895be952cff06d137a21c827f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.300522 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.300478    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.306041 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.306018    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:01.336076 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.336033    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.436524 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.436483    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.537024 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.536937    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.615350 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.615320    2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 23:54:01.637928 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.637901    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.658242 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.658212    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 24 23:54:01.658879 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.658399    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 24 23:54:01.658879 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.658399    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 24 23:54:01.734588 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.734557    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 24 23:54:01.738958 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.738931    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.744177 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.744150    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 24 23:54:01.759721 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.759671    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-23 23:49:00 +0000 UTC" deadline="2027-12-03 15:30:24.083027756 +0000 UTC"
Apr 24 23:54:01.759721 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.759715    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14103h36m22.323316198s"
Apr 24 23:54:01.769359 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.769331    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-68t7r"
Apr 24 23:54:01.775757 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.775735    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-68t7r"
Apr 24 23:54:01.816792 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:01.816576    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20602652b465b1663ffa2d71aa406f85.slice/crio-0cc32ae3f4def37ceffa162f1a6491cec0adca1363e54f1bdf2d3c950a61e536 WatchSource:0}: Error finding container 0cc32ae3f4def37ceffa162f1a6491cec0adca1363e54f1bdf2d3c950a61e536: Status 404 returned error can't find the container with id 0cc32ae3f4def37ceffa162f1a6491cec0adca1363e54f1bdf2d3c950a61e536
Apr 24 23:54:01.816983 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:01.816968    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bd46895be952cff06d137a21c827f1.slice/crio-21684efde5aeaf9b1c171dcab21735f074ab342f798faf572d55f4d784b8f7f2 WatchSource:0}: Error finding container 21684efde5aeaf9b1c171dcab21735f074ab342f798faf572d55f4d784b8f7f2: Status 404 returned error can't find the container with id 21684efde5aeaf9b1c171dcab21735f074ab342f798faf572d55f4d784b8f7f2
Apr 24 23:54:01.821086 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.821070    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:54:01.839305 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.839281    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:01.872638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.872570    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal" event={"ID":"52bd46895be952cff06d137a21c827f1","Type":"ContainerStarted","Data":"21684efde5aeaf9b1c171dcab21735f074ab342f798faf572d55f4d784b8f7f2"}
Apr 24 23:54:01.873510 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:01.873485    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal" event={"ID":"20602652b465b1663ffa2d71aa406f85","Type":"ContainerStarted","Data":"0cc32ae3f4def37ceffa162f1a6491cec0adca1363e54f1bdf2d3c950a61e536"}
Apr 24 23:54:01.939850 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:01.939816    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:02.040278 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.040247    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-82.ec2.internal\" not found"
Apr 24 23:54:02.064562 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.064496    2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 23:54:02.135338 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.135303    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:02.147260 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.147232    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 24 23:54:02.148711 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.148692    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal"
Apr 24 23:54:02.158630 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.158589    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 24 23:54:02.197438 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.197399    2576 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 23:54:02.602249 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.602218    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 23:54:02.717379 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.717341    2576 apiserver.go:52] "Watching apiserver"
Apr 24 23:54:02.724201 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.724173    2576 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 24 23:54:02.724647 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.724621    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f","openshift-dns/node-resolver-9jdgg","openshift-multus/multus-additional-cni-plugins-655v2","openshift-multus/multus-jrgm6","openshift-network-diagnostics/network-check-target-9ltqc","kube-system/konnectivity-agent-hrx4r","openshift-cluster-node-tuning-operator/tuned-jtpkf","openshift-image-registry/node-ca-d6fnc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal","openshift-multus/network-metrics-daemon-tcvmr","openshift-network-operator/iptables-alerter-pkr4p","openshift-ovn-kubernetes/ovnkube-node-9z2gl"]
Apr 24 23:54:02.727672 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.727652    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.729905 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.729883    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.730861 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.730843    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.731131 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.731111    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.732361 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.732137    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.732804 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.732782    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.732893 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.732831    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k2d9m\""
Apr 24 23:54:02.732953 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.732896    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2b6p8\""
Apr 24 23:54:02.733271 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.733199    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 24 23:54:02.734444 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.734427    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.736992 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.736973    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.739147 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.739131    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.741449 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.741429    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.743231 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysctl-d\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.743231 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-etc-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.743231 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-os-release\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.743231 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-multus-certs\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.743231 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-etc-kubernetes\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-device-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovnkube-script-lib\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-daemon-config\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysctl-conf\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fb4ead-406c-4a68-9fdd-bd950fced4a6-tmp-dir\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghn8\" (UniqueName: \"kubernetes.io/projected/70fb4ead-406c-4a68-9fdd-bd950fced4a6-kube-api-access-dghn8\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-ovn\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovnkube-config\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.743497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-cnibin\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfc4n\" (UniqueName: \"kubernetes.io/projected/fcc79ac7-41f1-46b6-b86b-a35706a53212-kube-api-access-jfc4n\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-sys\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-systemd\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743698    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.744050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.743700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-run\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-var-lib-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcc79ac7-41f1-46b6-b86b-a35706a53212-cni-binary-copy\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-modprobe-d\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-registration-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-cni-netd\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-kubelet\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-tuned\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-systemd-units\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptkf\" (UniqueName: \"kubernetes.io/projected/253f240c-2da1-4b34-ab6f-9696ac273b3c-kube-api-access-tptkf\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744340 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-host\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgjg\" (UniqueName: \"kubernetes.io/projected/348e2446-9bdf-4e08-8754-ab0d8c4fead4-kube-api-access-wcgjg\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-run-netns\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-node-log\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-cni-bin\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-env-overrides\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-socket-dir-parent\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-cni-bin\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-var-lib-kubelet\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-socket-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-log-socket\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-system-cni-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovn-node-metrics-cert\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.744766 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-netns\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70fb4ead-406c-4a68-9fdd-bd950fced4a6-hosts-file\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-cni-multus\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-hostroot\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysconfig\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-tmp\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.744994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-kubelet\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-conf-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-kubernetes\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-sys-fs\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-cni-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-lib-modules\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-systemd\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dltn\" (UniqueName: \"kubernetes.io/projected/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-kube-api-access-6dltn\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.745416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-slash\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.746017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.745271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-k8s-cni-cncf-io\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.746112 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.746094    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.746234 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.746212    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.746318 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.746272    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.748561 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.748543    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:02.748677 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.748645    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:02.748734 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.748647    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:02.748734 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.748710    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:02.749318 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.749298    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.749414 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.749326    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5cdl\""
Apr 24 23:54:02.749414 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.749379    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 24 23:54:02.749972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.749952    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 24 23:54:02.750504 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.750484    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r5w2g\""
Apr 24 23:54:02.751152 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.750890    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.752882 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.752863    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.752966 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.752903    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.753026 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.753005    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 24 23:54:02.753205 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.753185    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.753806    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.754432    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.754644    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.754808    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.754839    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755064    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755114    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-56vnv\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755389    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dzpsh\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755543    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 24 23:54:02.755622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755584    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7ts4\""
Apr 24 23:54:02.756073 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755677    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 24 23:54:02.756073 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.755717    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 24 23:54:02.756171 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.756077    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 24 23:54:02.756171 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.756079    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 24 23:54:02.756527 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.756510    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 24 23:54:02.756623 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.756527    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 24 23:54:02.756623 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.756510    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g77b2\""
Apr 24 23:54:02.756825 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.756805    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w8c42\""
Apr 24 23:54:02.776725 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.776701    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-23 23:49:01 +0000 UTC" deadline="2027-11-12 20:32:57.045908273 +0000 UTC"
Apr 24 23:54:02.776725 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.776724    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13604h38m54.26918687s"
Apr 24 23:54:02.836703 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.836670    2576 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 24 23:54:02.845923 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.845895    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-sys\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.845923 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.845929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42whv\" (UniqueName: \"kubernetes.io/projected/0cba9d02-9ab8-40ba-a254-3ee4430edef7-kube-api-access-42whv\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.845951    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-systemd\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.845977    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-run\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846001    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-var-lib-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-sys\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846040    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcc79ac7-41f1-46b6-b86b-a35706a53212-cni-binary-copy\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846048    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-systemd\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846068    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-modprobe-d\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846073    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-var-lib-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846049    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-run\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846173    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-registration-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846178    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-modprobe-d\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846191    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-cni-netd\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846207    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-kubelet\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1923eed-7e79-485f-8f7a-a5c7799e6a40-host-slash\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-registration-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846239    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-tuned\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-cni-netd\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846282    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-systemd-units\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-systemd-units\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846326    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tptkf\" (UniqueName: \"kubernetes.io/projected/253f240c-2da1-4b34-ab6f-9696ac273b3c-kube-api-access-tptkf\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-kubelet\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846453    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-host\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846483    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgjg\" (UniqueName: \"kubernetes.io/projected/348e2446-9bdf-4e08-8754-ab0d8c4fead4-kube-api-access-wcgjg\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.846639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846509    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-run-netns\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846532    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-host\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846541    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-node-log\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846554    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846571    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-run-netns\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846574    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-cni-bin\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846618    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-cni-bin\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846578    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-node-log\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846637    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-env-overrides\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcc79ac7-41f1-46b6-b86b-a35706a53212-cni-binary-copy\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-system-cni-dir\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec46b4ce-5ee5-4d29-8d14-8e3071852804-konnectivity-ca\") pod \"konnectivity-agent-hrx4r\" (UID: \"ec46b4ce-5ee5-4d29-8d14-8e3071852804\") " pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846726    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-socket-dir-parent\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-socket-dir-parent\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846793    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-cni-bin\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846819    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-var-lib-kubelet\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846839    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-socket-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846862    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-log-socket\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.847416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-cni-bin\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-var-lib-kubelet\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846888    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-log-socket\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846932    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-system-cni-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846949    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovn-node-metrics-cert\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846958    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-socket-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.846979    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-netns\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-system-cni-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847033    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70fb4ead-406c-4a68-9fdd-bd950fced4a6-hosts-file\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847062    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-netns\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847077    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-cni-multus\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847103    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-hostroot\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70fb4ead-406c-4a68-9fdd-bd950fced4a6-hosts-file\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-env-overrides\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.848017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-var-lib-cni-multus\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-hostroot\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847148    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847168    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysconfig\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847213    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-tmp\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysconfig\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847250    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847276    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-kubelet\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847299    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-conf-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-cnibin\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847338    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847350    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-kubernetes\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-kubelet\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-conf-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847420    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-kubernetes\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.849022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847481    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-sys-fs\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-sys-fs\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847569    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-cni-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnhx\" (UniqueName: \"kubernetes.io/projected/e44bd199-2f3d-41c4-9044-8635f902289e-kube-api-access-4vnhx\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777z8\" (UniqueName: \"kubernetes.io/projected/5b867a9a-b211-492e-a7c9-6081e587a486-kube-api-access-777z8\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-cni-dir\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847673    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-lib-modules\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847715    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-systemd\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-cni-binary-copy\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847834    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-systemd\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-lib-modules\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1923eed-7e79-485f-8f7a-a5c7799e6a40-iptables-alerter-script\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.847984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848027    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6dltn\" (UniqueName: \"kubernetes.io/projected/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-kube-api-access-6dltn\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848051    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-slash\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848072    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-k8s-cni-cncf-io\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-k8s-cni-cncf-io\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.849827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848155    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-slash\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848217    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysctl-d\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848274    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-etc-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848301    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-os-release\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-os-release\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848360    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-multus-certs\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-etc-openvswitch\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848378    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysctl-d\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848391    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-etc-kubernetes\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848423    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-device-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848452    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-os-release\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-host-run-multus-certs\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848452    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovnkube-script-lib\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848496    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-daemon-config\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848504    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-etc-kubernetes\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec46b4ce-5ee5-4d29-8d14-8e3071852804-agent-certs\") pod \"konnectivity-agent-hrx4r\" (UID: \"ec46b4ce-5ee5-4d29-8d14-8e3071852804\") " pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:02.850485 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848568    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysctl-conf\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848614    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fb4ead-406c-4a68-9fdd-bd950fced4a6-tmp-dir\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848638    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dghn8\" (UniqueName: \"kubernetes.io/projected/70fb4ead-406c-4a68-9fdd-bd950fced4a6-kube-api-access-dghn8\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848657    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-ovn\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848679    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848700    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovnkube-config\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0cba9d02-9ab8-40ba-a254-3ee4430edef7-serviceca\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848746    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-cnibin\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848771    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jfc4n\" (UniqueName: \"kubernetes.io/projected/fcc79ac7-41f1-46b6-b86b-a35706a53212-kube-api-access-jfc4n\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cba9d02-9ab8-40ba-a254-3ee4430edef7-host\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgnf\" (UniqueName: \"kubernetes.io/projected/b1923eed-7e79-485f-8f7a-a5c7799e6a40-kube-api-access-trgnf\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849040    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovnkube-script-lib\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fcc79ac7-41f1-46b6-b86b-a35706a53212-multus-daemon-config\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.848545    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/348e2446-9bdf-4e08-8754-ab0d8c4fead4-device-dir\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcc79ac7-41f1-46b6-b86b-a35706a53212-cnibin\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-sysctl-conf\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849382    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.850972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/253f240c-2da1-4b34-ab6f-9696ac273b3c-run-ovn\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.851417 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849692    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fb4ead-406c-4a68-9fdd-bd950fced4a6-tmp-dir\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.851417 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.849951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovnkube-config\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.851417 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.850631    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/253f240c-2da1-4b34-ab6f-9696ac273b3c-ovn-node-metrics-cert\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.851417 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.850811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-etc-tuned\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.851417 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.850831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-tmp\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.858170 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.857836    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptkf\" (UniqueName: \"kubernetes.io/projected/253f240c-2da1-4b34-ab6f-9696ac273b3c-kube-api-access-tptkf\") pod \"ovnkube-node-9z2gl\" (UID: \"253f240c-2da1-4b34-ab6f-9696ac273b3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:02.861104 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.860035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dltn\" (UniqueName: \"kubernetes.io/projected/9ecf1404-4f33-4307-aa17-5ab982d3a0f4-kube-api-access-6dltn\") pod \"tuned-jtpkf\" (UID: \"9ecf1404-4f33-4307-aa17-5ab982d3a0f4\") " pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:02.862425 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.862399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfc4n\" (UniqueName: \"kubernetes.io/projected/fcc79ac7-41f1-46b6-b86b-a35706a53212-kube-api-access-jfc4n\") pod \"multus-jrgm6\" (UID: \"fcc79ac7-41f1-46b6-b86b-a35706a53212\") " pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:02.863153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.863127    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghn8\" (UniqueName: \"kubernetes.io/projected/70fb4ead-406c-4a68-9fdd-bd950fced4a6-kube-api-access-dghn8\") pod \"node-resolver-9jdgg\" (UID: \"70fb4ead-406c-4a68-9fdd-bd950fced4a6\") " pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:02.863335 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.863320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgjg\" (UniqueName: \"kubernetes.io/projected/348e2446-9bdf-4e08-8754-ab0d8c4fead4-kube-api-access-wcgjg\") pod \"aws-ebs-csi-driver-node-jcc7f\" (UID: \"348e2446-9bdf-4e08-8754-ab0d8c4fead4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:02.949723 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949686    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-cni-binary-copy\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.949723 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949733    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1923eed-7e79-485f-8f7a-a5c7799e6a40-iptables-alerter-script\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949757    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949790    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-os-release\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949818    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec46b4ce-5ee5-4d29-8d14-8e3071852804-agent-certs\") pod \"konnectivity-agent-hrx4r\" (UID: \"ec46b4ce-5ee5-4d29-8d14-8e3071852804\") " pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949840    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949873    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0cba9d02-9ab8-40ba-a254-3ee4430edef7-serviceca\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949898    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cba9d02-9ab8-40ba-a254-3ee4430edef7-host\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949922    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-trgnf\" (UniqueName: \"kubernetes.io/projected/b1923eed-7e79-485f-8f7a-a5c7799e6a40-kube-api-access-trgnf\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.949967 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949948    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-42whv\" (UniqueName: \"kubernetes.io/projected/0cba9d02-9ab8-40ba-a254-3ee4430edef7-kube-api-access-42whv\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949973    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1923eed-7e79-485f-8f7a-a5c7799e6a40-host-slash\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.949994    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950010    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950041    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-system-cni-dir\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950066    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec46b4ce-5ee5-4d29-8d14-8e3071852804-konnectivity-ca\") pod \"konnectivity-agent-hrx4r\" (UID: \"ec46b4ce-5ee5-4d29-8d14-8e3071852804\") " pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950103    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950133    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-cnibin\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950165    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnhx\" (UniqueName: \"kubernetes.io/projected/e44bd199-2f3d-41c4-9044-8635f902289e-kube-api-access-4vnhx\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.950355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950191    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-777z8\" (UniqueName: \"kubernetes.io/projected/5b867a9a-b211-492e-a7c9-6081e587a486-kube-api-access-777z8\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:02.950762 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.950610    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:02.950762 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.950698    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:03.450676536 +0000 UTC m=+3.075716212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:02.950930 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1923eed-7e79-485f-8f7a-a5c7799e6a40-iptables-alerter-script\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.950930 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950840    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1923eed-7e79-485f-8f7a-a5c7799e6a40-host-slash\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.951029 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.950929    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-os-release\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951146    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cba9d02-9ab8-40ba-a254-3ee4430edef7-host\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951173    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-cni-binary-copy\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951406    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-system-cni-dir\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e44bd199-2f3d-41c4-9044-8635f902289e-cnibin\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951653    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e44bd199-2f3d-41c4-9044-8635f902289e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0cba9d02-9ab8-40ba-a254-3ee4430edef7-serviceca\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.951777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.951739    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec46b4ce-5ee5-4d29-8d14-8e3071852804-konnectivity-ca\") pod \"konnectivity-agent-hrx4r\" (UID: \"ec46b4ce-5ee5-4d29-8d14-8e3071852804\") " pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.953642 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.953623    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec46b4ce-5ee5-4d29-8d14-8e3071852804-agent-certs\") pod \"konnectivity-agent-hrx4r\" (UID: \"ec46b4ce-5ee5-4d29-8d14-8e3071852804\") " pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:02.958169 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.958146    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:02.958169 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.958173    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:02.958326 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.958188    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:02.958326 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:02.958274    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:03.458254833 +0000 UTC m=+3.083294508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:02.960520 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.960496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-777z8\" (UniqueName: \"kubernetes.io/projected/5b867a9a-b211-492e-a7c9-6081e587a486-kube-api-access-777z8\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:02.960640 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.960502    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42whv\" (UniqueName: \"kubernetes.io/projected/0cba9d02-9ab8-40ba-a254-3ee4430edef7-kube-api-access-42whv\") pod \"node-ca-d6fnc\" (UID: \"0cba9d02-9ab8-40ba-a254-3ee4430edef7\") " pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:02.960775 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.960756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgnf\" (UniqueName: \"kubernetes.io/projected/b1923eed-7e79-485f-8f7a-a5c7799e6a40-kube-api-access-trgnf\") pod \"iptables-alerter-pkr4p\" (UID: \"b1923eed-7e79-485f-8f7a-a5c7799e6a40\") " pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:02.961022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:02.960996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnhx\" (UniqueName: \"kubernetes.io/projected/e44bd199-2f3d-41c4-9044-8635f902289e-kube-api-access-4vnhx\") pod \"multus-additional-cni-plugins-655v2\" (UID: \"e44bd199-2f3d-41c4-9044-8635f902289e\") " pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:03.039529 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.039495    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtpkf"
Apr 24 23:54:03.048307 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.048277    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f"
Apr 24 23:54:03.060035 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.060012    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jdgg"
Apr 24 23:54:03.063724 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.063708    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:03.070224 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.070198    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jrgm6"
Apr 24 23:54:03.077818 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.077798    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pkr4p"
Apr 24 23:54:03.084387 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.084365    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:03.089986 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.089967    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d6fnc"
Apr 24 23:54:03.094413 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.094397    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-655v2"
Apr 24 23:54:03.426513 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:03.426484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1923eed_7e79_485f_8f7a_a5c7799e6a40.slice/crio-4877dc9b58f2dc5bbc0190d3cb2bc63c4d540d9aec0ebd8730e3b7731ae72962 WatchSource:0}: Error finding container 4877dc9b58f2dc5bbc0190d3cb2bc63c4d540d9aec0ebd8730e3b7731ae72962: Status 404 returned error can't find the container with id 4877dc9b58f2dc5bbc0190d3cb2bc63c4d540d9aec0ebd8730e3b7731ae72962
Apr 24 23:54:03.431828 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:03.431805    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec46b4ce_5ee5_4d29_8d14_8e3071852804.slice/crio-4f4f0fc97d618c7960a85f14f2748505d512e6ccd823d3192cabf6f87718f165 WatchSource:0}: Error finding container 4f4f0fc97d618c7960a85f14f2748505d512e6ccd823d3192cabf6f87718f165: Status 404 returned error can't find the container with id 4f4f0fc97d618c7960a85f14f2748505d512e6ccd823d3192cabf6f87718f165
Apr 24 23:54:03.432768 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:03.432737    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecf1404_4f33_4307_aa17_5ab982d3a0f4.slice/crio-cf51cdf9c0eb802c758e8004e04a5da68627c9df1c3230bd9dcf3d2255c27e47 WatchSource:0}: Error finding container cf51cdf9c0eb802c758e8004e04a5da68627c9df1c3230bd9dcf3d2255c27e47: Status 404 returned error can't find the container with id cf51cdf9c0eb802c758e8004e04a5da68627c9df1c3230bd9dcf3d2255c27e47
Apr 24 23:54:03.433564 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:03.433541    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44bd199_2f3d_41c4_9044_8635f902289e.slice/crio-e29b8a236ac39f55c8064f664562473a03bf6eaaf39052d90c4bd8706722c19f WatchSource:0}: Error finding container e29b8a236ac39f55c8064f664562473a03bf6eaaf39052d90c4bd8706722c19f: Status 404 returned error can't find the container with id e29b8a236ac39f55c8064f664562473a03bf6eaaf39052d90c4bd8706722c19f
Apr 24 23:54:03.435051 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:03.435027    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253f240c_2da1_4b34_ab6f_9696ac273b3c.slice/crio-264d3ce01db7c0194e1cf39f28979df6a17ffce04b6cd5b470d8c3edb0a29ec9 WatchSource:0}: Error finding container 264d3ce01db7c0194e1cf39f28979df6a17ffce04b6cd5b470d8c3edb0a29ec9: Status 404 returned error can't find the container with id 264d3ce01db7c0194e1cf39f28979df6a17ffce04b6cd5b470d8c3edb0a29ec9
Apr 24 23:54:03.435571 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:03.435500    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348e2446_9bdf_4e08_8754_ab0d8c4fead4.slice/crio-16caafc00e5f3d3f51ba82ab2836906e76d56ab10df5f6be6dc07e8612431402 WatchSource:0}: Error finding container 16caafc00e5f3d3f51ba82ab2836906e76d56ab10df5f6be6dc07e8612431402: Status 404 returned error can't find the container with id 16caafc00e5f3d3f51ba82ab2836906e76d56ab10df5f6be6dc07e8612431402
Apr 24 23:54:03.454732 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.454704    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:03.454848 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.454816    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:03.454898 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.454864    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:04.454850052 +0000 UTC m=+4.079889705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:03.556048 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.556002    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:03.556182 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.556168    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:03.556234 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.556189    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:03.556234 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.556200    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:03.556294 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.556255    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:04.556240584 +0000 UTC m=+4.181280243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:03.777218 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.777123    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-23 23:49:01 +0000 UTC" deadline="2027-09-27 09:51:07.372579186 +0000 UTC"
Apr 24 23:54:03.777218 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.777157    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12489h57m3.595425631s"
Apr 24 23:54:03.872620 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.869718    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:03.872620 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:03.869852    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:03.892396 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.892271    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"264d3ce01db7c0194e1cf39f28979df6a17ffce04b6cd5b470d8c3edb0a29ec9"}
Apr 24 23:54:03.897143 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.897084    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hrx4r" event={"ID":"ec46b4ce-5ee5-4d29-8d14-8e3071852804","Type":"ContainerStarted","Data":"4f4f0fc97d618c7960a85f14f2748505d512e6ccd823d3192cabf6f87718f165"}
Apr 24 23:54:03.901130 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.901078    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jdgg" event={"ID":"70fb4ead-406c-4a68-9fdd-bd950fced4a6","Type":"ContainerStarted","Data":"d28ed1bbde41b87ad68247fd8447fac9062e048ea3b319364f05f1d23ec434cc"}
Apr 24 23:54:03.903802 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.903774    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerStarted","Data":"e29b8a236ac39f55c8064f664562473a03bf6eaaf39052d90c4bd8706722c19f"}
Apr 24 23:54:03.906036 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.906010    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtpkf" event={"ID":"9ecf1404-4f33-4307-aa17-5ab982d3a0f4","Type":"ContainerStarted","Data":"cf51cdf9c0eb802c758e8004e04a5da68627c9df1c3230bd9dcf3d2255c27e47"}
Apr 24 23:54:03.908348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.908321    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d6fnc" event={"ID":"0cba9d02-9ab8-40ba-a254-3ee4430edef7","Type":"ContainerStarted","Data":"85661016e438f761bc5069c288f44e00a4a0228c2c84de1a64640947cfa2f68a"}
Apr 24 23:54:03.909798 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.909753    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkr4p" event={"ID":"b1923eed-7e79-485f-8f7a-a5c7799e6a40","Type":"ContainerStarted","Data":"4877dc9b58f2dc5bbc0190d3cb2bc63c4d540d9aec0ebd8730e3b7731ae72962"}
Apr 24 23:54:03.918614 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.916368    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal" event={"ID":"20602652b465b1663ffa2d71aa406f85","Type":"ContainerStarted","Data":"8edda3d3f6cea25fe18e46f4645d7c526affb90fc2f50cbf5889e7a7424ac9e7"}
Apr 24 23:54:03.921531 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.921472    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jrgm6" event={"ID":"fcc79ac7-41f1-46b6-b86b-a35706a53212","Type":"ContainerStarted","Data":"32c0b32d90d0f5b73298d9acb013c54465163c688d4ff994ae8e7571f70b8a91"}
Apr 24 23:54:03.930653 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:03.923720    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f" event={"ID":"348e2446-9bdf-4e08-8754-ab0d8c4fead4","Type":"ContainerStarted","Data":"16caafc00e5f3d3f51ba82ab2836906e76d56ab10df5f6be6dc07e8612431402"}
Apr 24 23:54:04.346207 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.346179    2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 23:54:04.463260 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.463219    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:04.463433 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.463381    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:04.463481 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.463450    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:06.463430818 +0000 UTC m=+6.088470476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:04.564089 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.563983    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:04.564262 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.564154    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:04.564262 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.564174    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:04.564262 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.564186    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:04.564262 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.564243    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:06.564224503 +0000 UTC m=+6.189264161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:04.869903 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.869823    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:04.870410 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:04.869960    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:04.937271 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.936895    2576 generic.go:358] "Generic (PLEG): container finished" podID="52bd46895be952cff06d137a21c827f1" containerID="ce61ce1b6f9e613596ff6e9615be3c61660677d12c1e3e8ccf9769e12ff07761" exitCode=0
Apr 24 23:54:04.937271 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.937014    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal" event={"ID":"52bd46895be952cff06d137a21c827f1","Type":"ContainerDied","Data":"ce61ce1b6f9e613596ff6e9615be3c61660677d12c1e3e8ccf9769e12ff07761"}
Apr 24 23:54:04.954094 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:04.954036    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-82.ec2.internal" podStartSLOduration=2.9540161 podStartE2EDuration="2.9540161s" podCreationTimestamp="2026-04-24 23:54:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:54:03.944373444 +0000 UTC m=+3.569413131" watchObservedRunningTime="2026-04-24 23:54:04.9540161 +0000 UTC m=+4.579055778"
Apr 24 23:54:05.870125 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:05.869887    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:05.870588 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:05.870248    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:05.949835 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:05.949799    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal" event={"ID":"52bd46895be952cff06d137a21c827f1","Type":"ContainerStarted","Data":"e6930aade9be0d2b37a1ec2f299fefa6348d371c9c03a54ca62a75a01242ea93"}
Apr 24 23:54:05.963432 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:05.963372    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-82.ec2.internal" podStartSLOduration=3.963354168 podStartE2EDuration="3.963354168s" podCreationTimestamp="2026-04-24 23:54:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:54:05.963112485 +0000 UTC m=+5.588152160" watchObservedRunningTime="2026-04-24 23:54:05.963354168 +0000 UTC m=+5.588393845"
Apr 24 23:54:06.481381 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:06.480724    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:06.481381 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.480928    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:06.481381 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.480995    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:10.480974605 +0000 UTC m=+10.106014263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:06.582350 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:06.581696    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:06.582350 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.581879    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:06.582350 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.581897    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:06.582350 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.581910    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:06.582350 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.581967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:10.581948937 +0000 UTC m=+10.206988608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:06.869902 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:06.869823    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:06.870032 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:06.869929    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:07.869201 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:07.869164    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:07.869570 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:07.869311    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:08.869754 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:08.869719    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:08.870151 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:08.869873    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:09.869473 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:09.869401    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:09.869673 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:09.869637    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:10.515488 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:10.515448    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:10.515933 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.515647    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:10.515933 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.515710    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:18.515690835 +0000 UTC m=+18.140730497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:10.616131 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:10.616103    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:10.616291 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.616269    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:10.616291 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.616289    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:10.616460 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.616303    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:10.616460 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.616371    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:18.616351931 +0000 UTC m=+18.241391601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:10.870491 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:10.870403    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:10.870662 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:10.870551    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:11.869093 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:11.869058    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:11.869593 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:11.869210    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:12.869968 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:12.869920    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:12.870363 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:12.870052    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:13.869301 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:13.869257    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:13.869489 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:13.869390    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:14.869796 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:14.869766    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:14.870315 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:14.869874    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:15.869725 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:15.869687    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:15.869922 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:15.869824    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:16.869641 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:16.869584    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:16.869804 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:16.869738    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:17.869615 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:17.869569    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:17.870032 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:17.869718    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:18.580861 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:18.580825    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:18.581041 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.581017    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:18.581122 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.581106    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:34.581084362 +0000 UTC m=+34.206124021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:18.681682 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:18.681642    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:18.681848 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.681804    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:18.681848 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.681831    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:18.681848 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.681845    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:18.681974 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.681907    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:54:34.68189118 +0000 UTC m=+34.306930835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:18.870104 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:18.870021    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:18.870548 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:18.870149    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:19.869654 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:19.869626    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:19.869806 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:19.869748    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:20.869819 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:20.869792    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:20.870246 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:20.869872    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:21.869747 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.869713    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:21.869926 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:21.869846    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:21.979655 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.979410    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jrgm6" event={"ID":"fcc79ac7-41f1-46b6-b86b-a35706a53212","Type":"ContainerStarted","Data":"1c1dade6000791392d4ab0effbd882a0a70e4eea13dace0ecf40bd99b29bcf1e"}
Apr 24 23:54:21.980778 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.980746    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f" event={"ID":"348e2446-9bdf-4e08-8754-ab0d8c4fead4","Type":"ContainerStarted","Data":"4bb96feeeda8330fc594b0016d8bd605a0a7e955acf4455d04cde32a9ac8b1b7"}
Apr 24 23:54:21.983142 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983122    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 24 23:54:21.983458 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983437    2576 generic.go:358] "Generic (PLEG): container finished" podID="253f240c-2da1-4b34-ab6f-9696ac273b3c" containerID="df0575f558f1f50fb5e6143ac7057d6dbf8339f381e2660872b1b6d69a8d5f37" exitCode=1
Apr 24 23:54:21.983529 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983507    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"e3f0e2c7d9b7965e49cabac17dde62756b88701aeff19fd98f359f846ac1f770"}
Apr 24 23:54:21.983572 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983542    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"7c9bed5acfd16397debd63ec5f1bcb33973595f9636e95f58c56d0db4ade5861"}
Apr 24 23:54:21.983572 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983557    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"ab2b9276377acbfb6175b9f4478959f12e75facef7052b3aa2e7c4590bb9acd6"}
Apr 24 23:54:21.983572 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983572    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"f7dd54382217ec29dae25ad86a5956ce2d77c1fd086444addbe2aaf7fd2586ab"}
Apr 24 23:54:21.983717 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983584    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerDied","Data":"df0575f558f1f50fb5e6143ac7057d6dbf8339f381e2660872b1b6d69a8d5f37"}
Apr 24 23:54:21.983717 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.983615    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"93be86431c1a64d1c8ec26929c3b219997f6fc2062563b14bb4052f2080485f7"}
Apr 24 23:54:21.984623 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.984585    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hrx4r" event={"ID":"ec46b4ce-5ee5-4d29-8d14-8e3071852804","Type":"ContainerStarted","Data":"402386413da6a9301e39bb43bd8fd6cbd60078e119bad6c1aee7e1ceeeffe8bb"}
Apr 24 23:54:21.985792 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.985772    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jdgg" event={"ID":"70fb4ead-406c-4a68-9fdd-bd950fced4a6","Type":"ContainerStarted","Data":"5c5af2247ea17f7507d4e9f69fa53e27c7199174ed7611eee80c68633db9efe7"}
Apr 24 23:54:21.987061 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.987043    2576 generic.go:358] "Generic (PLEG): container finished" podID="e44bd199-2f3d-41c4-9044-8635f902289e" containerID="6af1963b900d7c95024bd6a917babb14474ea0557bbf6e08a3bfea5eafbadadb" exitCode=0
Apr 24 23:54:21.987153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.987069    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerDied","Data":"6af1963b900d7c95024bd6a917babb14474ea0557bbf6e08a3bfea5eafbadadb"}
Apr 24 23:54:21.988355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.988283    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtpkf" event={"ID":"9ecf1404-4f33-4307-aa17-5ab982d3a0f4","Type":"ContainerStarted","Data":"b5c4894bc0a7d6d379e7c3095a33da284bb3d11709f2d35338cb6b65399f13a8"}
Apr 24 23:54:21.989407 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:21.989389    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d6fnc" event={"ID":"0cba9d02-9ab8-40ba-a254-3ee4430edef7","Type":"ContainerStarted","Data":"84287e1dacad4fd131d5e440534450aeb10316fa307dc4e8ce839246ab08e214"}
Apr 24 23:54:22.002341 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.002305    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-jrgm6" podStartSLOduration=4.535350579 podStartE2EDuration="22.002294573s" podCreationTimestamp="2026-04-24 23:54:00 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.439930403 +0000 UTC m=+3.064970060" lastFinishedPulling="2026-04-24 23:54:20.906874386 +0000 UTC m=+20.531914054" observedRunningTime="2026-04-24 23:54:22.001210054 +0000 UTC m=+21.626249740" watchObservedRunningTime="2026-04-24 23:54:22.002294573 +0000 UTC m=+21.627334228"
Apr 24 23:54:22.020714 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.020669    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-9jdgg" podStartSLOduration=4.7433554319999995 podStartE2EDuration="22.020656001s" podCreationTimestamp="2026-04-24 23:54:00 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.439157968 +0000 UTC m=+3.064197624" lastFinishedPulling="2026-04-24 23:54:20.716458525 +0000 UTC m=+20.341498193" observedRunningTime="2026-04-24 23:54:22.02050019 +0000 UTC m=+21.645539865" watchObservedRunningTime="2026-04-24 23:54:22.020656001 +0000 UTC m=+21.645695678"
Apr 24 23:54:22.068583 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.068529    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d6fnc" podStartSLOduration=3.782996206 podStartE2EDuration="21.068513997s" podCreationTimestamp="2026-04-24 23:54:01 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.431262622 +0000 UTC m=+3.056302281" lastFinishedPulling="2026-04-24 23:54:20.716780403 +0000 UTC m=+20.341820072" observedRunningTime="2026-04-24 23:54:22.044424195 +0000 UTC m=+21.669463871" watchObservedRunningTime="2026-04-24 23:54:22.068513997 +0000 UTC m=+21.693553672"
Apr 24 23:54:22.087915 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.087865    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hrx4r" podStartSLOduration=3.805136182 podStartE2EDuration="21.087850835s" podCreationTimestamp="2026-04-24 23:54:01 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.433735437 +0000 UTC m=+3.058775093" lastFinishedPulling="2026-04-24 23:54:20.716450075 +0000 UTC m=+20.341489746" observedRunningTime="2026-04-24 23:54:22.087834012 +0000 UTC m=+21.712873703" watchObservedRunningTime="2026-04-24 23:54:22.087850835 +0000 UTC m=+21.712890512"
Apr 24 23:54:22.088059 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.087943    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jtpkf" podStartSLOduration=4.806466859 podStartE2EDuration="22.087938592s" podCreationTimestamp="2026-04-24 23:54:00 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.434999621 +0000 UTC m=+3.060039276" lastFinishedPulling="2026-04-24 23:54:20.716471356 +0000 UTC m=+20.341511009" observedRunningTime="2026-04-24 23:54:22.068631555 +0000 UTC m=+21.693671231" watchObservedRunningTime="2026-04-24 23:54:22.087938592 +0000 UTC m=+21.712978265"
Apr 24 23:54:22.484440 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.484413    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 24 23:54:22.820618 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.820482    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-24T23:54:22.484433788Z","UUID":"a3a17d84-a954-4ad8-87d8-18ba5c11c0df","Handler":null,"Name":"","Endpoint":""}
Apr 24 23:54:22.822289 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.822244    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 24 23:54:22.822289 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.822283    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 24 23:54:22.869427 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.869388    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:22.869653 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:22.869527    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:22.993498 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.993450    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pkr4p" event={"ID":"b1923eed-7e79-485f-8f7a-a5c7799e6a40","Type":"ContainerStarted","Data":"6d1d2d95e317e1e7b6d7e330acc3edc389808bae0144578364947577b99448f1"}
Apr 24 23:54:22.995840 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:22.995788    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f" event={"ID":"348e2446-9bdf-4e08-8754-ab0d8c4fead4","Type":"ContainerStarted","Data":"b3fa3549afeb06b9413dad142f4d98739158f4f5d40f9741a82289a3bc985bde"}
Apr 24 23:54:23.010250 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:23.009644    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pkr4p" podStartSLOduration=5.532651792 podStartE2EDuration="23.00962564s" podCreationTimestamp="2026-04-24 23:54:00 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.429095746 +0000 UTC m=+3.054135403" lastFinishedPulling="2026-04-24 23:54:20.906069583 +0000 UTC m=+20.531109251" observedRunningTime="2026-04-24 23:54:23.00948064 +0000 UTC m=+22.634520317" watchObservedRunningTime="2026-04-24 23:54:23.00962564 +0000 UTC m=+22.634665319"
Apr 24 23:54:23.400020 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:23.399994    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jdgg_70fb4ead-406c-4a68-9fdd-bd950fced4a6/dns-node-resolver/0.log"
Apr 24 23:54:23.869160 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:23.869132    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:23.869295 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:23.869273    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:24.000342 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:23.999641    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f" event={"ID":"348e2446-9bdf-4e08-8754-ab0d8c4fead4","Type":"ContainerStarted","Data":"352f395f2a3383e7603174a44bb6e8d81ca273aa288034cbffcb0307cc21bbda"}
Apr 24 23:54:24.004777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:24.004754    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 24 23:54:24.005181 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:24.005148    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"ee4772231518e3bfb368969e577eea9d11364304726cb9654fad856000824db7"}
Apr 24 23:54:24.027215 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:24.027159    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcc7f" podStartSLOduration=3.9588169779999998 podStartE2EDuration="24.027139653s" podCreationTimestamp="2026-04-24 23:54:00 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.438344001 +0000 UTC m=+3.063383654" lastFinishedPulling="2026-04-24 23:54:23.506666675 +0000 UTC m=+23.131706329" observedRunningTime="2026-04-24 23:54:24.026561084 +0000 UTC m=+23.651600794" watchObservedRunningTime="2026-04-24 23:54:24.027139653 +0000 UTC m=+23.652179327"
Apr 24 23:54:24.581616 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:24.581573    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d6fnc_0cba9d02-9ab8-40ba-a254-3ee4430edef7/node-ca/0.log"
Apr 24 23:54:24.869507 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:24.869424    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:24.869700 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:24.869533    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:25.869654 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:25.869617    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:25.870229 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:25.869761    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:26.229785 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:26.229752    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:26.230513 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:26.230484    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:26.869746 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:26.869710    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:26.870280 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:26.869813    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:26.965789 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:26.965761    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:26.966322 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:26.966297    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hrx4r"
Apr 24 23:54:27.013155 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.013125    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 24 23:54:27.013495 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.013468    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"270e7fcd99fa3761abf68efb16f27668f0a87a93d460821edd40a31766de5a8f"}
Apr 24 23:54:27.013804 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.013787    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:27.013981 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.013968    2576 scope.go:117] "RemoveContainer" containerID="df0575f558f1f50fb5e6143ac7057d6dbf8339f381e2660872b1b6d69a8d5f37"
Apr 24 23:54:27.015553 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.015532    2576 generic.go:358] "Generic (PLEG): container finished" podID="e44bd199-2f3d-41c4-9044-8635f902289e" containerID="c5d274a4ff7091c029c01dbb049428853a16b8be23bfd5f6590b7779088d6d61" exitCode=0
Apr 24 23:54:27.015682 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.015637    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerDied","Data":"c5d274a4ff7091c029c01dbb049428853a16b8be23bfd5f6590b7779088d6d61"}
Apr 24 23:54:27.030466 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.030445    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:27.869909 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:27.869818    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:27.870255 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:27.869931    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:28.020545 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.020515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 24 23:54:28.020861 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.020839    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" event={"ID":"253f240c-2da1-4b34-ab6f-9696ac273b3c","Type":"ContainerStarted","Data":"3361c2140e0c9c0a5adbce3b03a081b4464ca68a122960b7dbfb5404f67f455f"}
Apr 24 23:54:28.021065 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.021047    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:28.021137 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.021079    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:28.022842 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.022817    2576 generic.go:358] "Generic (PLEG): container finished" podID="e44bd199-2f3d-41c4-9044-8635f902289e" containerID="65502a68924aedb7b7d68d3aa3385959b57d2b9af1081fc05e354fc913d2a674" exitCode=0
Apr 24 23:54:28.022932 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.022889    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerDied","Data":"65502a68924aedb7b7d68d3aa3385959b57d2b9af1081fc05e354fc913d2a674"}
Apr 24 23:54:28.036701 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.036671    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:54:28.078415 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.078363    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl" podStartSLOduration=10.566753359 podStartE2EDuration="28.078348973s" podCreationTimestamp="2026-04-24 23:54:00 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.439539431 +0000 UTC m=+3.064579088" lastFinishedPulling="2026-04-24 23:54:20.951135035 +0000 UTC m=+20.576174702" observedRunningTime="2026-04-24 23:54:28.048474236 +0000 UTC m=+27.673513922" watchObservedRunningTime="2026-04-24 23:54:28.078348973 +0000 UTC m=+27.703388648"
Apr 24 23:54:28.870199 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:28.870102    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:28.870698 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:28.870205    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:29.028409 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:29.028376    2576 generic.go:358] "Generic (PLEG): container finished" podID="e44bd199-2f3d-41c4-9044-8635f902289e" containerID="ebf18af08f73b65054bb44ce10633c39d114005f97fc356ef6105a06ff961442" exitCode=0
Apr 24 23:54:29.028569 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:29.028474    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerDied","Data":"ebf18af08f73b65054bb44ce10633c39d114005f97fc356ef6105a06ff961442"}
Apr 24 23:54:29.869823 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:29.869786    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:29.870014 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:29.869909    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:30.870756 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:30.870106    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:30.870756 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:30.870263    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:31.869833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:31.869797    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:31.870052 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:31.869927    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:32.869635 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:32.869590    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:32.870096 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:32.869704    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:33.869161 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:33.869124    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:33.869479 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:33.869267    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:34.597931 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:34.597894    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:34.598306 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.598015    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:34.598306 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.598071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs podName:5b867a9a-b211-492e-a7c9-6081e587a486 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:06.598054497 +0000 UTC m=+66.223094165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs") pod "network-metrics-daemon-tcvmr" (UID: "5b867a9a-b211-492e-a7c9-6081e587a486") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 23:54:34.698868 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:34.698833    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:34.699026 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.698950    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 23:54:34.699026 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.698963    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 23:54:34.699026 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.698972    2576 projected.go:194] Error preparing data for projected volume kube-api-access-nvqbx for pod openshift-network-diagnostics/network-check-target-9ltqc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:34.699026 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.699019    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx podName:b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:06.69900607 +0000 UTC m=+66.324045724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nvqbx" (UniqueName: "kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx") pod "network-check-target-9ltqc" (UID: "b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 23:54:34.869442 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:34.869410    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:34.869572 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:34.869545    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:35.043941 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:35.043910    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerStarted","Data":"56ca2125d8485630854f69d24958e3a296cafce760c7c228c172430a377ae884"}
Apr 24 23:54:35.869743 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:35.869706    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:35.870211 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:35.869821    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:36.047497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:36.047461    2576 generic.go:358] "Generic (PLEG): container finished" podID="e44bd199-2f3d-41c4-9044-8635f902289e" containerID="56ca2125d8485630854f69d24958e3a296cafce760c7c228c172430a377ae884" exitCode=0
Apr 24 23:54:36.047677 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:36.047515    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerDied","Data":"56ca2125d8485630854f69d24958e3a296cafce760c7c228c172430a377ae884"}
Apr 24 23:54:36.869357 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:36.869320    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:36.869552 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:36.869440    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:37.051717 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:37.051681    2576 generic.go:358] "Generic (PLEG): container finished" podID="e44bd199-2f3d-41c4-9044-8635f902289e" containerID="380d3d5a9686e97ecee5115df5bc97522513e166a47ae119aaefee908c82998e" exitCode=0
Apr 24 23:54:37.052202 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:37.051727    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerDied","Data":"380d3d5a9686e97ecee5115df5bc97522513e166a47ae119aaefee908c82998e"}
Apr 24 23:54:37.869401 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:37.869361    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:37.869594 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:37.869472    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:38.055829 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:38.055796    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-655v2" event={"ID":"e44bd199-2f3d-41c4-9044-8635f902289e","Type":"ContainerStarted","Data":"27095eaf8e3afb80db10874cd33528ace065df9dece102bf0ee5cf064962ed12"}
Apr 24 23:54:38.089292 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:38.087629    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-655v2" podStartSLOduration=5.748008343 podStartE2EDuration="37.08757992s" podCreationTimestamp="2026-04-24 23:54:01 +0000 UTC" firstStartedPulling="2026-04-24 23:54:03.435906437 +0000 UTC m=+3.060946095" lastFinishedPulling="2026-04-24 23:54:34.775478016 +0000 UTC m=+34.400517672" observedRunningTime="2026-04-24 23:54:38.085708678 +0000 UTC m=+37.710748354" watchObservedRunningTime="2026-04-24 23:54:38.08757992 +0000 UTC m=+37.712619597"
Apr 24 23:54:38.869569 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:38.869534    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:38.869754 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:38.869661    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:39.869471 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:39.869435    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:39.869938 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:39.869568    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:40.870398 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:40.870366    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:40.870846 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:40.870482    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:41.869437 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:41.869399    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:41.869639 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:41.869515    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:42.869834 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:42.869795    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:42.870322 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:42.869898    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:43.869749 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:43.869707    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:43.869930 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:43.869822    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:44.869316 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:44.869277    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:44.869512 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:44.869383    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:45.869682 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:45.869649    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:45.870114 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:45.869754    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:46.869288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:46.869259    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:46.869456 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:46.869352    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:47.870029 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:47.869995    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:47.870449 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:47.870111    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:48.870067 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:48.870027    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:48.870444 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:48.870131    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:49.869733 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:49.869691    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:49.869975 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:49.869809    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:50.879064 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:50.879029    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:50.879541 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:50.879128    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:51.260999 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:51.260763    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tcvmr"]
Apr 24 23:54:51.261146 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:51.261127    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:51.261265 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:51.261243    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:51.276375 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:51.276347    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9ltqc"]
Apr 24 23:54:51.276542 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:51.276459    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:51.276585 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:51.276535    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:52.869975 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:52.869944    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:52.870408 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:52.870071    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9ltqc" podUID="b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3"
Apr 24 23:54:52.870408 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:52.870117    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:52.870408 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:54:52.870214    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tcvmr" podUID="5b867a9a-b211-492e-a7c9-6081e587a486"
Apr 24 23:54:54.683722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.683687    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-82.ec2.internal" event="NodeReady"
Apr 24 23:54:54.684103 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.683828    2576 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 24 23:54:54.727802 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.727765    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-86bbr"]
Apr 24 23:54:54.778315 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.778227    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l5jkt"]
Apr 24 23:54:54.778493 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.778417    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.780870 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.780843    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 24 23:54:54.781014 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.780853    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 24 23:54:54.781014 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.781002    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 24 23:54:54.781334 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.781315    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gprwq\""
Apr 24 23:54:54.781380 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.781348    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 24 23:54:54.795741 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.795676    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-86bbr"]
Apr 24 23:54:54.795741 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.795702    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l5jkt"]
Apr 24 23:54:54.795919 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.795807    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.797926 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.797906    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xmfk4\""
Apr 24 23:54:54.798085 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.798067    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 24 23:54:54.798154 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.798073    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 24 23:54:54.835316 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835285    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8dxqr"]
Apr 24 23:54:54.835461 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqs9\" (UniqueName: \"kubernetes.io/projected/89554b8a-f951-4924-8546-b51462423669-kube-api-access-cvqs9\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.835523 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89554b8a-f951-4924-8546-b51462423669-data-volume\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.835523 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlw78\" (UniqueName: \"kubernetes.io/projected/67d24f30-2b79-44ee-bf51-10b4444a5035-kube-api-access-wlw78\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.835523 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89554b8a-f951-4924-8546-b51462423669-crio-socket\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.835676 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89554b8a-f951-4924-8546-b51462423669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.835676 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d24f30-2b79-44ee-bf51-10b4444a5035-tmp-dir\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.835676 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89554b8a-f951-4924-8546-b51462423669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.835676 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67d24f30-2b79-44ee-bf51-10b4444a5035-config-volume\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.835804 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.835675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67d24f30-2b79-44ee-bf51-10b4444a5035-metrics-tls\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.854944 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.854913    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8dxqr"]
Apr 24 23:54:54.855092 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.854974    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:54.857577 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.857554    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 24 23:54:54.857737 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.857665    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 24 23:54:54.857737 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.857679    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 24 23:54:54.857833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.857742    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mn69n\""
Apr 24 23:54:54.869839 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.869819    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:54:54.870038 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.870018    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:54:54.871989 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.871965    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 24 23:54:54.871989 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.871985    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 24 23:54:54.872150 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.872026    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8lkgq\""
Apr 24 23:54:54.872150 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.872133    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9ssg\""
Apr 24 23:54:54.872256 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.872148    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 24 23:54:54.936412 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936379    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67d24f30-2b79-44ee-bf51-10b4444a5035-config-volume\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.936412 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11529be1-47c8-4118-a866-36eb412239c8-cert\") pod \"ingress-canary-8dxqr\" (UID: \"11529be1-47c8-4118-a866-36eb412239c8\") " pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:54.936664 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwk4\" (UniqueName: \"kubernetes.io/projected/11529be1-47c8-4118-a866-36eb412239c8-kube-api-access-4gwk4\") pod \"ingress-canary-8dxqr\" (UID: \"11529be1-47c8-4118-a866-36eb412239c8\") " pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:54.936664 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936517    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67d24f30-2b79-44ee-bf51-10b4444a5035-metrics-tls\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.936664 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936554    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqs9\" (UniqueName: \"kubernetes.io/projected/89554b8a-f951-4924-8546-b51462423669-kube-api-access-cvqs9\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.936775 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936723    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89554b8a-f951-4924-8546-b51462423669-data-volume\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.936942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936911    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlw78\" (UniqueName: \"kubernetes.io/projected/67d24f30-2b79-44ee-bf51-10b4444a5035-kube-api-access-wlw78\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.937023 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936954    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89554b8a-f951-4924-8546-b51462423669-crio-socket\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.937023 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.936981    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89554b8a-f951-4924-8546-b51462423669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.937023 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89554b8a-f951-4924-8546-b51462423669-data-volume\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.937193 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67d24f30-2b79-44ee-bf51-10b4444a5035-config-volume\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.937193 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937036    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d24f30-2b79-44ee-bf51-10b4444a5035-tmp-dir\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.937193 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89554b8a-f951-4924-8546-b51462423669-crio-socket\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.937193 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937099    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89554b8a-f951-4924-8546-b51462423669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.937401 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d24f30-2b79-44ee-bf51-10b4444a5035-tmp-dir\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.937553 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.937536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89554b8a-f951-4924-8546-b51462423669-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.940813 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.940784    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89554b8a-f951-4924-8546-b51462423669-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:54.940939 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.940909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67d24f30-2b79-44ee-bf51-10b4444a5035-metrics-tls\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.944712 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.944618    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlw78\" (UniqueName: \"kubernetes.io/projected/67d24f30-2b79-44ee-bf51-10b4444a5035-kube-api-access-wlw78\") pod \"dns-default-l5jkt\" (UID: \"67d24f30-2b79-44ee-bf51-10b4444a5035\") " pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:54.944712 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:54.944655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqs9\" (UniqueName: \"kubernetes.io/projected/89554b8a-f951-4924-8546-b51462423669-kube-api-access-cvqs9\") pod \"insights-runtime-extractor-86bbr\" (UID: \"89554b8a-f951-4924-8546-b51462423669\") " pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:55.037415 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.037381    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11529be1-47c8-4118-a866-36eb412239c8-cert\") pod \"ingress-canary-8dxqr\" (UID: \"11529be1-47c8-4118-a866-36eb412239c8\") " pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:55.037565 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.037455    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwk4\" (UniqueName: \"kubernetes.io/projected/11529be1-47c8-4118-a866-36eb412239c8-kube-api-access-4gwk4\") pod \"ingress-canary-8dxqr\" (UID: \"11529be1-47c8-4118-a866-36eb412239c8\") " pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:55.039934 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.039914    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11529be1-47c8-4118-a866-36eb412239c8-cert\") pod \"ingress-canary-8dxqr\" (UID: \"11529be1-47c8-4118-a866-36eb412239c8\") " pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:55.050677 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.049407    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwk4\" (UniqueName: \"kubernetes.io/projected/11529be1-47c8-4118-a866-36eb412239c8-kube-api-access-4gwk4\") pod \"ingress-canary-8dxqr\" (UID: \"11529be1-47c8-4118-a866-36eb412239c8\") " pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:55.098300 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.098268    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-86bbr"
Apr 24 23:54:55.113120 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.113090    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:55.163969 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.163932    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8dxqr"
Apr 24 23:54:55.275782 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.275750    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l5jkt"]
Apr 24 23:54:55.279230 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:55.279206    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d24f30_2b79_44ee_bf51_10b4444a5035.slice/crio-9a22ba3bccf299c3adcf35132a928c9bc03653acfe4c8d64b98abde5c5ffc141 WatchSource:0}: Error finding container 9a22ba3bccf299c3adcf35132a928c9bc03653acfe4c8d64b98abde5c5ffc141: Status 404 returned error can't find the container with id 9a22ba3bccf299c3adcf35132a928c9bc03653acfe4c8d64b98abde5c5ffc141
Apr 24 23:54:55.293528 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.293504    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-86bbr"]
Apr 24 23:54:55.296645 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:55.296614    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89554b8a_f951_4924_8546_b51462423669.slice/crio-d96fccfdc2e8de9d687aad9f58cc7653c5d13079dccd5525ca14fa0fa664eb88 WatchSource:0}: Error finding container d96fccfdc2e8de9d687aad9f58cc7653c5d13079dccd5525ca14fa0fa664eb88: Status 404 returned error can't find the container with id d96fccfdc2e8de9d687aad9f58cc7653c5d13079dccd5525ca14fa0fa664eb88
Apr 24 23:54:55.303509 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:55.303455    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8dxqr"]
Apr 24 23:54:55.306383 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:55.306360    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11529be1_47c8_4118_a866_36eb412239c8.slice/crio-43d887c083591e10ab84456778abad769a28595b9854186677758d0f8674c112 WatchSource:0}: Error finding container 43d887c083591e10ab84456778abad769a28595b9854186677758d0f8674c112: Status 404 returned error can't find the container with id 43d887c083591e10ab84456778abad769a28595b9854186677758d0f8674c112
Apr 24 23:54:56.088089 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:56.088048    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8dxqr" event={"ID":"11529be1-47c8-4118-a866-36eb412239c8","Type":"ContainerStarted","Data":"43d887c083591e10ab84456778abad769a28595b9854186677758d0f8674c112"}
Apr 24 23:54:56.090050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:56.090021    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5jkt" event={"ID":"67d24f30-2b79-44ee-bf51-10b4444a5035","Type":"ContainerStarted","Data":"9a22ba3bccf299c3adcf35132a928c9bc03653acfe4c8d64b98abde5c5ffc141"}
Apr 24 23:54:56.091775 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:56.091746    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbr" event={"ID":"89554b8a-f951-4924-8546-b51462423669","Type":"ContainerStarted","Data":"78386cb3d7fbcd27e0630486c75fb5a5b89dfed0d136e36b3b29334c6e7ff838"}
Apr 24 23:54:56.091893 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:56.091782    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbr" event={"ID":"89554b8a-f951-4924-8546-b51462423669","Type":"ContainerStarted","Data":"d96fccfdc2e8de9d687aad9f58cc7653c5d13079dccd5525ca14fa0fa664eb88"}
Apr 24 23:54:58.097735 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.097693    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8dxqr" event={"ID":"11529be1-47c8-4118-a866-36eb412239c8","Type":"ContainerStarted","Data":"426bb3c993dca186705ebd6e290daae78b0690277f8b5bb36ca9cd54b795ee3c"}
Apr 24 23:54:58.099554 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.099521    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5jkt" event={"ID":"67d24f30-2b79-44ee-bf51-10b4444a5035","Type":"ContainerStarted","Data":"74fa7b0ceffe0e551a45fae835fdb630b798c545bd06465b7377c8f5adbc4ff8"}
Apr 24 23:54:58.099708 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.099556    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5jkt" event={"ID":"67d24f30-2b79-44ee-bf51-10b4444a5035","Type":"ContainerStarted","Data":"4dcc4201f832427d6d0d7ff8312259fbaa8ba9828e4b42eba72368b98bacae43"}
Apr 24 23:54:58.099708 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.099640    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:54:58.101431 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.101396    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbr" event={"ID":"89554b8a-f951-4924-8546-b51462423669","Type":"ContainerStarted","Data":"bedb6929c95c4e6d8055b9a0acc14b373310f7e9228741efb0addf1dd23bcce5"}
Apr 24 23:54:58.113218 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.113163    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8dxqr" podStartSLOduration=1.859568434 podStartE2EDuration="4.113144904s" podCreationTimestamp="2026-04-24 23:54:54 +0000 UTC" firstStartedPulling="2026-04-24 23:54:55.307999631 +0000 UTC m=+54.933039285" lastFinishedPulling="2026-04-24 23:54:57.561576101 +0000 UTC m=+57.186615755" observedRunningTime="2026-04-24 23:54:58.112006772 +0000 UTC m=+57.737046448" watchObservedRunningTime="2026-04-24 23:54:58.113144904 +0000 UTC m=+57.738184583"
Apr 24 23:54:58.128228 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.128170    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-l5jkt" podStartSLOduration=1.8476320290000001 podStartE2EDuration="4.128148254s" podCreationTimestamp="2026-04-24 23:54:54 +0000 UTC" firstStartedPulling="2026-04-24 23:54:55.281057882 +0000 UTC m=+54.906097536" lastFinishedPulling="2026-04-24 23:54:57.561574104 +0000 UTC m=+57.186613761" observedRunningTime="2026-04-24 23:54:58.126878901 +0000 UTC m=+57.751918577" watchObservedRunningTime="2026-04-24 23:54:58.128148254 +0000 UTC m=+57.753187930"
Apr 24 23:54:58.824618 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.824562    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-68b8f44ddb-9fh26"]
Apr 24 23:54:58.828948 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.828919    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.831089 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.831066    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Apr 24 23:54:58.831204 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.831113    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Apr 24 23:54:58.831204 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.831123    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Apr 24 23:54:58.832163 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.832055    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Apr 24 23:54:58.832239 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.832187    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Apr 24 23:54:58.832239 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.832231    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w2hp6\""
Apr 24 23:54:58.832307 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.832187    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Apr 24 23:54:58.832725 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.832708    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Apr 24 23:54:58.837254 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.837236    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Apr 24 23:54:58.838218 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.837931    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b8f44ddb-9fh26"]
Apr 24 23:54:58.875245 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-config\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.875245 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-oauth-config\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.875431 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-service-ca\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.875431 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-trusted-ca-bundle\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.875431 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-serving-cert\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.875431 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhg8h\" (UniqueName: \"kubernetes.io/projected/1614041e-bf4b-4117-92ef-bfdd7a9543f7-kube-api-access-xhg8h\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.875431 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.875413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-oauth-serving-cert\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.976579 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976552    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-trusted-ca-bundle\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.976722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976612    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-serving-cert\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.976722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976635    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xhg8h\" (UniqueName: \"kubernetes.io/projected/1614041e-bf4b-4117-92ef-bfdd7a9543f7-kube-api-access-xhg8h\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.976722 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976662    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-oauth-serving-cert\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.976883 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976861    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-config\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.976939 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976906    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-oauth-config\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.977012 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.976990    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-service-ca\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.977451 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.977431    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-trusted-ca-bundle\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.977534 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.977503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-config\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.977618 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.977580    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-oauth-serving-cert\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.977671 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.977580    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-service-ca\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.979700 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.979681    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-serving-cert\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.979781 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.979723    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-oauth-config\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:58.984846 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:58.984816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhg8h\" (UniqueName: \"kubernetes.io/projected/1614041e-bf4b-4117-92ef-bfdd7a9543f7-kube-api-access-xhg8h\") pod \"console-68b8f44ddb-9fh26\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") " pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:59.020559 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.020538    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5676c8c784-2st66"]
Apr 24 23:54:59.024763 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.024749    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.027110 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.027091    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 24 23:54:59.027370 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.027352    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 24 23:54:59.027788 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.027770    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tbnbn\""
Apr 24 23:54:59.027890 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.027803    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Apr 24 23:54:59.027890 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.027872    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 24 23:54:59.027890 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.027880    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Apr 24 23:54:59.034050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.034029    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5676c8c784-2st66"]
Apr 24 23:54:59.077995 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.077967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fe51d1e-6b60-455a-aace-77aea96906e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.078132 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.078016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/8fe51d1e-6b60-455a-aace-77aea96906e0-kube-api-access-zfr4f\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.078132 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.078061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe51d1e-6b60-455a-aace-77aea96906e0-metrics-client-ca\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.078132 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.078082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fe51d1e-6b60-455a-aace-77aea96906e0-prometheus-operator-tls\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.107972 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.107940    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86bbr" event={"ID":"89554b8a-f951-4924-8546-b51462423669","Type":"ContainerStarted","Data":"a342e3a1d5773225e7cdbd918085d752ee5daf05b4c3a5e4b3f68698f5db20ce"}
Apr 24 23:54:59.123984 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.123932    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-86bbr" podStartSLOduration=1.62356742 podStartE2EDuration="5.123919353s" podCreationTimestamp="2026-04-24 23:54:54 +0000 UTC" firstStartedPulling="2026-04-24 23:54:55.406688511 +0000 UTC m=+55.031728180" lastFinishedPulling="2026-04-24 23:54:58.907040446 +0000 UTC m=+58.532080113" observedRunningTime="2026-04-24 23:54:59.123868784 +0000 UTC m=+58.748908461" watchObservedRunningTime="2026-04-24 23:54:59.123919353 +0000 UTC m=+58.748959029"
Apr 24 23:54:59.140903 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.140874    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:54:59.178852 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.178817    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe51d1e-6b60-455a-aace-77aea96906e0-metrics-client-ca\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.179004 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.178869    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fe51d1e-6b60-455a-aace-77aea96906e0-prometheus-operator-tls\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.179004 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.178949    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fe51d1e-6b60-455a-aace-77aea96906e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.179004 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.178994    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/8fe51d1e-6b60-455a-aace-77aea96906e0-kube-api-access-zfr4f\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.180117 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.179870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fe51d1e-6b60-455a-aace-77aea96906e0-metrics-client-ca\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.181905 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.181883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fe51d1e-6b60-455a-aace-77aea96906e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.182570 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.182547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fe51d1e-6b60-455a-aace-77aea96906e0-prometheus-operator-tls\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.188523 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.188476    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfr4f\" (UniqueName: \"kubernetes.io/projected/8fe51d1e-6b60-455a-aace-77aea96906e0-kube-api-access-zfr4f\") pod \"prometheus-operator-5676c8c784-2st66\" (UID: \"8fe51d1e-6b60-455a-aace-77aea96906e0\") " pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.255419 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.255382    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b8f44ddb-9fh26"]
Apr 24 23:54:59.258648 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:59.258591    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1614041e_bf4b_4117_92ef_bfdd7a9543f7.slice/crio-6401604a315ae12c64dd8dec892051e9f893a19e549509978b62f80873631a4b WatchSource:0}: Error finding container 6401604a315ae12c64dd8dec892051e9f893a19e549509978b62f80873631a4b: Status 404 returned error can't find the container with id 6401604a315ae12c64dd8dec892051e9f893a19e549509978b62f80873631a4b
Apr 24 23:54:59.333590 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.333503    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66"
Apr 24 23:54:59.447845 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:54:59.447813    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5676c8c784-2st66"]
Apr 24 23:54:59.451091 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:54:59.451060    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe51d1e_6b60_455a_aace_77aea96906e0.slice/crio-5db89e0a83dbfba366787f2bf6fcedd4725476dc72865c832c4acd229c2610c7 WatchSource:0}: Error finding container 5db89e0a83dbfba366787f2bf6fcedd4725476dc72865c832c4acd229c2610c7: Status 404 returned error can't find the container with id 5db89e0a83dbfba366787f2bf6fcedd4725476dc72865c832c4acd229c2610c7
Apr 24 23:55:00.044055 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:00.044026    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9z2gl"
Apr 24 23:55:00.112407 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:00.112373    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b8f44ddb-9fh26" event={"ID":"1614041e-bf4b-4117-92ef-bfdd7a9543f7","Type":"ContainerStarted","Data":"6401604a315ae12c64dd8dec892051e9f893a19e549509978b62f80873631a4b"}
Apr 24 23:55:00.114268 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:00.114219    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66" event={"ID":"8fe51d1e-6b60-455a-aace-77aea96906e0","Type":"ContainerStarted","Data":"5db89e0a83dbfba366787f2bf6fcedd4725476dc72865c832c4acd229c2610c7"}
Apr 24 23:55:01.119018 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:01.118974    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66" event={"ID":"8fe51d1e-6b60-455a-aace-77aea96906e0","Type":"ContainerStarted","Data":"06d63ddbf793e8f0e3d5b4bc94ff1c71ef8a7e4b1e40bbfeb368b6c24d4b492a"}
Apr 24 23:55:01.119018 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:01.119023    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66" event={"ID":"8fe51d1e-6b60-455a-aace-77aea96906e0","Type":"ContainerStarted","Data":"eb4b6203dea39cc06f253f610340303852acda693dfdd3c9f85db181a6a6599d"}
Apr 24 23:55:01.138701 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:01.138646    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5676c8c784-2st66" podStartSLOduration=1.021713404 podStartE2EDuration="2.1386291s" podCreationTimestamp="2026-04-24 23:54:59 +0000 UTC" firstStartedPulling="2026-04-24 23:54:59.453070928 +0000 UTC m=+59.078110583" lastFinishedPulling="2026-04-24 23:55:00.569986611 +0000 UTC m=+60.195026279" observedRunningTime="2026-04-24 23:55:01.137826823 +0000 UTC m=+60.762866502" watchObservedRunningTime="2026-04-24 23:55:01.1386291 +0000 UTC m=+60.763668777"
Apr 24 23:55:02.122706 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:02.122611    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b8f44ddb-9fh26" event={"ID":"1614041e-bf4b-4117-92ef-bfdd7a9543f7","Type":"ContainerStarted","Data":"ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1"}
Apr 24 23:55:02.139658 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:02.139585    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-68b8f44ddb-9fh26" podStartSLOduration=1.571079549 podStartE2EDuration="4.139571144s" podCreationTimestamp="2026-04-24 23:54:58 +0000 UTC" firstStartedPulling="2026-04-24 23:54:59.260554398 +0000 UTC m=+58.885594055" lastFinishedPulling="2026-04-24 23:55:01.829045997 +0000 UTC m=+61.454085650" observedRunningTime="2026-04-24 23:55:02.138560229 +0000 UTC m=+61.763599909" watchObservedRunningTime="2026-04-24 23:55:02.139571144 +0000 UTC m=+61.764610820"
Apr 24 23:55:03.378620 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.378574    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"]
Apr 24 23:55:03.403337 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.403306    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"]
Apr 24 23:55:03.403470 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.403429    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.405834 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.405811    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Apr 24 23:55:03.405951 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.405821    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Apr 24 23:55:03.405951 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.405934    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rw4kq\""
Apr 24 23:55:03.430083 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.430051    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"]
Apr 24 23:55:03.451622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.451571    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zqb26"]
Apr 24 23:55:03.451761 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.451727    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.454210 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.454190    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-62lnn\""
Apr 24 23:55:03.454735 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.454717    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Apr 24 23:55:03.454826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.454811    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Apr 24 23:55:03.455016 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.455005    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Apr 24 23:55:03.475647 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.475623    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"]
Apr 24 23:55:03.475739 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.475704    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.477763 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.477742    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 24 23:55:03.478028 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.478006    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 24 23:55:03.478028 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.478013    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 24 23:55:03.478168 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.478037    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6rtvs\""
Apr 24 23:55:03.511743 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.511743 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.511925 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.511925 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.511925 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.511925 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.511925 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.511925 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-accelerators-collector-config\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512107 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.511945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-wtmp\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512107 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-textfile\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512107 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jr6j\" (UniqueName: \"kubernetes.io/projected/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-kube-api-access-4jr6j\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512107 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9b7b\" (UniqueName: \"kubernetes.io/projected/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-api-access-l9b7b\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.512107 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-root\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.512348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-tls\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.512348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-metrics-client-ca\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-sys\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.512348 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.512287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpg2b\" (UniqueName: \"kubernetes.io/projected/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-kube-api-access-fpg2b\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.612969 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.612914    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-wtmp\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.612969 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.612963    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-textfile\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.612969 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.612985    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4jr6j\" (UniqueName: \"kubernetes.io/projected/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-kube-api-access-4jr6j\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613007    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9b7b\" (UniqueName: \"kubernetes.io/projected/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-api-access-l9b7b\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613027    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-root\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613046    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613064    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-tls\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613089    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613113    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-metrics-client-ca\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-wtmp\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613145    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-sys\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-root\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:03.613204    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:03.613226    2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:03.613273    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-tls podName:778fb23e-9606-41ce-a9a7-ba09b3ecbfb3 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:04.113251864 +0000 UTC m=+63.738291522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-tls") pod "node-exporter-zqb26" (UID: "778fb23e-9606-41ce-a9a7-ba09b3ecbfb3") : secret "node-exporter-tls" not found
Apr 24 23:55:03.613288 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:03.613294    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-tls podName:54c60e27-d809-4a9b-9f8c-fa8c15b9bf34 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:04.113285786 +0000 UTC m=+63.738325446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-tls") pod "kube-state-metrics-69db897b98-qnfzz" (UID: "54c60e27-d809-4a9b-9f8c-fa8c15b9bf34") : secret "kube-state-metrics-tls" not found
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613314    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpg2b\" (UniqueName: \"kubernetes.io/projected/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-kube-api-access-fpg2b\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-sys\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613389    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613440    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-textfile\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613491    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613521    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613580    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613670    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613706    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613734    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-accelerators-collector-config\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.613942 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.613887    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-metrics-client-ca\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.614490 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.614214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.614490 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.614278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-accelerators-collector-config\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.614698 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.614677    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.614844 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.614821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.614957 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.614909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.616133 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.616110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.616344 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.616316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.616509 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.616489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.616628 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.616587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.624181 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.624137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpg2b\" (UniqueName: \"kubernetes.io/projected/db99497f-4c14-45c0-bfc0-7d8e6c70cee8-kube-api-access-fpg2b\") pod \"openshift-state-metrics-9d44df66c-59prq\" (UID: \"db99497f-4c14-45c0-bfc0-7d8e6c70cee8\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.624295 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.624278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jr6j\" (UniqueName: \"kubernetes.io/projected/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-kube-api-access-4jr6j\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:03.624392 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.624375    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9b7b\" (UniqueName: \"kubernetes.io/projected/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-api-access-l9b7b\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:03.711729 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.711646    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"
Apr 24 23:55:03.835846 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:03.835814    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-59prq"]
Apr 24 23:55:03.867564 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:03.867534    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb99497f_4c14_45c0_bfc0_7d8e6c70cee8.slice/crio-7f174ba25e1cbc3b38070bbf6022e581201cf8dce831dbb8e6d03a29c16e8a1b WatchSource:0}: Error finding container 7f174ba25e1cbc3b38070bbf6022e581201cf8dce831dbb8e6d03a29c16e8a1b: Status 404 returned error can't find the container with id 7f174ba25e1cbc3b38070bbf6022e581201cf8dce831dbb8e6d03a29c16e8a1b
Apr 24 23:55:04.117661 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.117629    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-tls\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:04.117827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.117670    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:04.120302 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.120273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54c60e27-d809-4a9b-9f8c-fa8c15b9bf34-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-qnfzz\" (UID: \"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:04.120449 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.120431    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/778fb23e-9606-41ce-a9a7-ba09b3ecbfb3-node-exporter-tls\") pod \"node-exporter-zqb26\" (UID: \"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3\") " pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:04.129748 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.129722    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq" event={"ID":"db99497f-4c14-45c0-bfc0-7d8e6c70cee8","Type":"ContainerStarted","Data":"36c5e823f3b5e57024f4e6174908b53d3104bcd59c92fd07b1c68a3a2e57a625"}
Apr 24 23:55:04.129855 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.129753    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq" event={"ID":"db99497f-4c14-45c0-bfc0-7d8e6c70cee8","Type":"ContainerStarted","Data":"319fcffc5fb634b4042955b559f1cfdac074d0afc7131b64ccc6e9e2196dc1cd"}
Apr 24 23:55:04.129855 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.129763    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq" event={"ID":"db99497f-4c14-45c0-bfc0-7d8e6c70cee8","Type":"ContainerStarted","Data":"7f174ba25e1cbc3b38070bbf6022e581201cf8dce831dbb8e6d03a29c16e8a1b"}
Apr 24 23:55:04.361048 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.361009    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"
Apr 24 23:55:04.383769 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.383741    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zqb26"
Apr 24 23:55:04.391551 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:04.391518    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778fb23e_9606_41ce_a9a7_ba09b3ecbfb3.slice/crio-3eb5a5593f44774856441986cc9dc785d70788706349d737f984bedad0b2d0bf WatchSource:0}: Error finding container 3eb5a5593f44774856441986cc9dc785d70788706349d737f984bedad0b2d0bf: Status 404 returned error can't find the container with id 3eb5a5593f44774856441986cc9dc785d70788706349d737f984bedad0b2d0bf
Apr 24 23:55:04.454466 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.454437    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 24 23:55:04.470709 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.470683    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.473183 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473156    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Apr 24 23:55:04.473323 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473206    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Apr 24 23:55:04.473323 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473217    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Apr 24 23:55:04.473323 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473226    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Apr 24 23:55:04.473563 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473545    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Apr 24 23:55:04.473635 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473613    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Apr 24 23:55:04.473681 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473661    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Apr 24 23:55:04.473903 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473885    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bptk9\""
Apr 24 23:55:04.474002 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473946    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Apr 24 23:55:04.474002 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.473955    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Apr 24 23:55:04.476700 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.476662    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 24 23:55:04.491798 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.491774    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-qnfzz"]
Apr 24 23:55:04.494573 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:04.494545    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c60e27_d809_4a9b_9f8c_fa8c15b9bf34.slice/crio-0b093f0866319c8325e94eb33a637543fab1ff8a0e069e8d79b7381b2fb80568 WatchSource:0}: Error finding container 0b093f0866319c8325e94eb33a637543fab1ff8a0e069e8d79b7381b2fb80568: Status 404 returned error can't find the container with id 0b093f0866319c8325e94eb33a637543fab1ff8a0e069e8d79b7381b2fb80568
Apr 24 23:55:04.520292 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-web-config\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2j44\" (UniqueName: \"kubernetes.io/projected/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-kube-api-access-m2j44\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520864 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520864 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520864 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520864 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.520864 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.521046 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.521046 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.520926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-config-out\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.621854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.621745    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.621854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.621796    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.621854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.621821    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.621854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.621847    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.621896    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:04.621961    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-trusted-ca-bundle podName:a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:05.121935493 +0000 UTC m=+64.746975163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54") : configmap references non-existent config key: ca-bundle.crt
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622001    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622024    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-config-out\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622050    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622073    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-web-config\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622153 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622114    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622470 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622183    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622470 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622210    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.622470 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.622252    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m2j44\" (UniqueName: \"kubernetes.io/projected/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-kube-api-access-m2j44\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.623211 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:04.623041    2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Apr 24 23:55:04.623211 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:04.623098    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-main-tls podName:a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:05.123080909 +0000 UTC m=+64.748120566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54") : secret "alertmanager-main-tls" not found
Apr 24 23:55:04.623868 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.623840    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.624500 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.624472    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.625434 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.625381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.626368 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.626061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.626368 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.626329    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.627574 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.627523    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.627574 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.627531    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-web-config\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.628073 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.628052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.628865 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.628839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-config-out\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.629238 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.629201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:04.635307 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:04.635281    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2j44\" (UniqueName: \"kubernetes.io/projected/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-kube-api-access-m2j44\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:05.126639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.126586    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:05.126813 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.126700    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:05.127432 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.127405    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:05.129490 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.129462    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:05.133626 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.133540    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz" event={"ID":"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34","Type":"ContainerStarted","Data":"0b093f0866319c8325e94eb33a637543fab1ff8a0e069e8d79b7381b2fb80568"}
Apr 24 23:55:05.134868 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.134841    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqb26" event={"ID":"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3","Type":"ContainerStarted","Data":"3eb5a5593f44774856441986cc9dc785d70788706349d737f984bedad0b2d0bf"}
Apr 24 23:55:05.382281 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.382199    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 24 23:55:05.854810 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:05.854772    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 24 23:55:05.858327 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:05.858293    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d0c0a8_9e99_47ca_bb69_c7fbfa816c54.slice/crio-47cbcb0c31ef212acad54a08281faa37303b2919ab4165d66e6a49247bd6836d WatchSource:0}: Error finding container 47cbcb0c31ef212acad54a08281faa37303b2919ab4165d66e6a49247bd6836d: Status 404 returned error can't find the container with id 47cbcb0c31ef212acad54a08281faa37303b2919ab4165d66e6a49247bd6836d
Apr 24 23:55:06.139938 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.139894    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqb26" event={"ID":"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3","Type":"ContainerStarted","Data":"99e3f05adcfd980ea15be8b73e9180093687dec21f6982d7f824a0b0e76f5d01"}
Apr 24 23:55:06.142276 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.142235    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq" event={"ID":"db99497f-4c14-45c0-bfc0-7d8e6c70cee8","Type":"ContainerStarted","Data":"b56757e587283c20cf8644971305ccba3f80abbc4bb881d1313a34c69b657112"}
Apr 24 23:55:06.143593 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.143559    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"47cbcb0c31ef212acad54a08281faa37303b2919ab4165d66e6a49247bd6836d"}
Apr 24 23:55:06.174546 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.174495    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-59prq" podStartSLOduration=1.529621221 podStartE2EDuration="3.174478923s" podCreationTimestamp="2026-04-24 23:55:03 +0000 UTC" firstStartedPulling="2026-04-24 23:55:04.063447474 +0000 UTC m=+63.688487135" lastFinishedPulling="2026-04-24 23:55:05.70830517 +0000 UTC m=+65.333344837" observedRunningTime="2026-04-24 23:55:06.173908259 +0000 UTC m=+65.798947935" watchObservedRunningTime="2026-04-24 23:55:06.174478923 +0000 UTC m=+65.799518599"
Apr 24 23:55:06.460974 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.460850    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"]
Apr 24 23:55:06.464920 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.464895    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.467125    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.467620    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7zkhm\""
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.467871    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.467993    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.467993    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-h8teimurp8p7\""
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.468026    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Apr 24 23:55:06.468236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.468115    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Apr 24 23:55:06.479479 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.477852    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"]
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-tls\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-metrics-client-ca\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-grpc-tls\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmxh\" (UniqueName: \"kubernetes.io/projected/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-kube-api-access-8pmxh\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.540384 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.540293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641228 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641129    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-tls\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641228 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641191    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641228 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641217    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-metrics-client-ca\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641228 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641236    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-grpc-tls\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641562 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641260    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmxh\" (UniqueName: \"kubernetes.io/projected/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-kube-api-access-8pmxh\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641562 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641296    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641562 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641319    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.641562 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641362    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:55:06.641562 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.641381    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.642669 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.642316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-metrics-client-ca\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.645222 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.644714    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.645222 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.644722    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-grpc-tls\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.645222 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.644815    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 24 23:55:06.645222 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.645183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-tls\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.646578 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.646523    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.646578 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.646570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.646887 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.646777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.649900 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.649877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmxh\" (UniqueName: \"kubernetes.io/projected/6e9a1e5d-0ba8-41db-bb50-31e22229e9b8-kube-api-access-8pmxh\") pod \"thanos-querier-5dbb5b4cc9-d9h9q\" (UID: \"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8\") " pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.655907 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.655880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b867a9a-b211-492e-a7c9-6081e587a486-metrics-certs\") pod \"network-metrics-daemon-tcvmr\" (UID: \"5b867a9a-b211-492e-a7c9-6081e587a486\") " pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:55:06.742469 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.742430    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:55:06.744868 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.744842    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 24 23:55:06.754974 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.754947    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 24 23:55:06.766806 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.766765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqbx\" (UniqueName: \"kubernetes.io/projected/b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3-kube-api-access-nvqbx\") pod \"network-check-target-9ltqc\" (UID: \"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3\") " pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:55:06.782948 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.782914    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:06.881395 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.881359    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8lkgq\""
Apr 24 23:55:06.885255 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.885225    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9ssg\""
Apr 24 23:55:06.890029 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.889996    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:55:06.893771 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:06.893709    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tcvmr"
Apr 24 23:55:07.138371 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.138343    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tcvmr"]
Apr 24 23:55:07.141300 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:07.141273    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b867a9a_b211_492e_a7c9_6081e587a486.slice/crio-d216265988888a489c625671718e51016f49cd5dbf202d359aa5bddc6360d94f WatchSource:0}: Error finding container d216265988888a489c625671718e51016f49cd5dbf202d359aa5bddc6360d94f: Status 404 returned error can't find the container with id d216265988888a489c625671718e51016f49cd5dbf202d359aa5bddc6360d94f
Apr 24 23:55:07.149282 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.149247    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz" event={"ID":"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34","Type":"ContainerStarted","Data":"41ebd5bd19a618be48204143130907c58ea976c36e539aca3987c43803543299"}
Apr 24 23:55:07.149372 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.149295    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz" event={"ID":"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34","Type":"ContainerStarted","Data":"45c058fb6bec134842c2992063ab38046426dbdc9d50d4c68330bbea18679e91"}
Apr 24 23:55:07.149372 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.149310    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz" event={"ID":"54c60e27-d809-4a9b-9f8c-fa8c15b9bf34","Type":"ContainerStarted","Data":"3ed58049572fc1912df44a925c9b5824afb4b0d5d7f805195ed557ae77ef6b83"}
Apr 24 23:55:07.151313 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.151289    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"636fd90aa610fa6cdbe44fba0daa63f6d9882e0b776984b7041aafca6c3e4322"}
Apr 24 23:55:07.152593 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.152541    2576 generic.go:358] "Generic (PLEG): container finished" podID="778fb23e-9606-41ce-a9a7-ba09b3ecbfb3" containerID="99e3f05adcfd980ea15be8b73e9180093687dec21f6982d7f824a0b0e76f5d01" exitCode=0
Apr 24 23:55:07.152708 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.152633    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqb26" event={"ID":"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3","Type":"ContainerDied","Data":"99e3f05adcfd980ea15be8b73e9180093687dec21f6982d7f824a0b0e76f5d01"}
Apr 24 23:55:07.154292 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.154272    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tcvmr" event={"ID":"5b867a9a-b211-492e-a7c9-6081e587a486","Type":"ContainerStarted","Data":"d216265988888a489c625671718e51016f49cd5dbf202d359aa5bddc6360d94f"}
Apr 24 23:55:07.166275 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.166223    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-69db897b98-qnfzz" podStartSLOduration=2.322868771 podStartE2EDuration="4.166207911s" podCreationTimestamp="2026-04-24 23:55:03 +0000 UTC" firstStartedPulling="2026-04-24 23:55:04.497498683 +0000 UTC m=+64.122538351" lastFinishedPulling="2026-04-24 23:55:06.340837822 +0000 UTC m=+65.965877491" observedRunningTime="2026-04-24 23:55:07.165583946 +0000 UTC m=+66.790623620" watchObservedRunningTime="2026-04-24 23:55:07.166207911 +0000 UTC m=+66.791247586"
Apr 24 23:55:07.363881 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.363849    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9ltqc"]
Apr 24 23:55:07.366675 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:07.366643    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54c5cf9_9b7c_4090_a9e4_e1aadbb64dc3.slice/crio-8d94d0b0a0819e3aa5cfc584e90853a56ed16af3034174cd944dbe0bdf7bdfa0 WatchSource:0}: Error finding container 8d94d0b0a0819e3aa5cfc584e90853a56ed16af3034174cd944dbe0bdf7bdfa0: Status 404 returned error can't find the container with id 8d94d0b0a0819e3aa5cfc584e90853a56ed16af3034174cd944dbe0bdf7bdfa0
Apr 24 23:55:07.371846 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.371823    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"]
Apr 24 23:55:07.374149 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:07.374126    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9a1e5d_0ba8_41db_bb50_31e22229e9b8.slice/crio-9db65da4b052137da537ebc08e1a34ca350854227632d9ade0ecb7a33ba2a460 WatchSource:0}: Error finding container 9db65da4b052137da537ebc08e1a34ca350854227632d9ade0ecb7a33ba2a460: Status 404 returned error can't find the container with id 9db65da4b052137da537ebc08e1a34ca350854227632d9ade0ecb7a33ba2a460
Apr 24 23:55:07.807232 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.807193    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5776cf486c-bmk6q"]
Apr 24 23:55:07.811047 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.811023    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.815498 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.815288    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9dtb0kvb4gmes\""
Apr 24 23:55:07.815931 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.815894    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 24 23:55:07.816351 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.816331    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l588s\""
Apr 24 23:55:07.816539 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.816523    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Apr 24 23:55:07.816757 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.816741    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Apr 24 23:55:07.817310 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.817296    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Apr 24 23:55:07.840742 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.840692    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5776cf486c-bmk6q"]
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9pz\" (UniqueName: \"kubernetes.io/projected/ad93192c-534f-4ab1-ae0d-091859d832b2-kube-api-access-fp9pz\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad93192c-534f-4ab1-ae0d-091859d832b2-audit-log\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-client-ca-bundle\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-secret-metrics-server-client-certs\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad93192c-534f-4ab1-ae0d-091859d832b2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad93192c-534f-4ab1-ae0d-091859d832b2-metrics-server-audit-profiles\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.856833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.856701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-secret-metrics-server-tls\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.957851 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.957807    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-secret-metrics-server-client-certs\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.958269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.957868    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad93192c-534f-4ab1-ae0d-091859d832b2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.958269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.957902    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad93192c-534f-4ab1-ae0d-091859d832b2-metrics-server-audit-profiles\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.958269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.957934    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-secret-metrics-server-tls\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.958269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.957991    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9pz\" (UniqueName: \"kubernetes.io/projected/ad93192c-534f-4ab1-ae0d-091859d832b2-kube-api-access-fp9pz\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.958269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.958036    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad93192c-534f-4ab1-ae0d-091859d832b2-audit-log\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.958269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.958060    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-client-ca-bundle\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.959575 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.959548    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad93192c-534f-4ab1-ae0d-091859d832b2-metrics-server-audit-profiles\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.959864 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.959838    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad93192c-534f-4ab1-ae0d-091859d832b2-audit-log\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.960117 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.960074    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad93192c-534f-4ab1-ae0d-091859d832b2-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.961174 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.961131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-client-ca-bundle\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.963258 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.963220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-secret-metrics-server-client-certs\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.965135 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.965083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad93192c-534f-4ab1-ae0d-091859d832b2-secret-metrics-server-tls\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:07.968190 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:07.968125    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9pz\" (UniqueName: \"kubernetes.io/projected/ad93192c-534f-4ab1-ae0d-091859d832b2-kube-api-access-fp9pz\") pod \"metrics-server-5776cf486c-bmk6q\" (UID: \"ad93192c-534f-4ab1-ae0d-091859d832b2\") " pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:08.110281 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.110239    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l5jkt"
Apr 24 23:55:08.126179 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.125792    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:08.164634 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.164552    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9ltqc" event={"ID":"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3","Type":"ContainerStarted","Data":"8d94d0b0a0819e3aa5cfc584e90853a56ed16af3034174cd944dbe0bdf7bdfa0"}
Apr 24 23:55:08.166174 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.166108    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"9db65da4b052137da537ebc08e1a34ca350854227632d9ade0ecb7a33ba2a460"}
Apr 24 23:55:08.168281 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.168248    2576 generic.go:358] "Generic (PLEG): container finished" podID="a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54" containerID="636fd90aa610fa6cdbe44fba0daa63f6d9882e0b776984b7041aafca6c3e4322" exitCode=0
Apr 24 23:55:08.168408 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.168338    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerDied","Data":"636fd90aa610fa6cdbe44fba0daa63f6d9882e0b776984b7041aafca6c3e4322"}
Apr 24 23:55:08.175337 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.175312    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqb26" event={"ID":"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3","Type":"ContainerStarted","Data":"07ee5cbfd74c1360f2f7900c28b0131dcbe7f2d952e67ca20306cc748c93624b"}
Apr 24 23:55:08.175545 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.175519    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zqb26" event={"ID":"778fb23e-9606-41ce-a9a7-ba09b3ecbfb3","Type":"ContainerStarted","Data":"2a9cf9d517d01607b3bc8cc88d02f0efdb242437d557b7f7f6844393f38bf2e0"}
Apr 24 23:55:08.187955 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.184915    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"]
Apr 24 23:55:08.192236 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.192212    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:08.193216 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.193174    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"]
Apr 24 23:55:08.196935 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.196497    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-dw6qf\""
Apr 24 23:55:08.196935 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.196670    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Apr 24 23:55:08.238621 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.238478    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zqb26" podStartSLOduration=3.921944699 podStartE2EDuration="5.238455126s" podCreationTimestamp="2026-04-24 23:55:03 +0000 UTC" firstStartedPulling="2026-04-24 23:55:04.393326595 +0000 UTC m=+64.018366260" lastFinishedPulling="2026-04-24 23:55:05.709837028 +0000 UTC m=+65.334876687" observedRunningTime="2026-04-24 23:55:08.236893996 +0000 UTC m=+67.861933673" watchObservedRunningTime="2026-04-24 23:55:08.238455126 +0000 UTC m=+67.863494857"
Apr 24 23:55:08.266147 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.263467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff43f34c-be72-4604-b7a1-be1ab3ba9ce4-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-n8z9m\" (UID: \"ff43f34c-be72-4604-b7a1-be1ab3ba9ce4\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:08.283888 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.283834    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-76599bc7bd-jsk4c"]
Apr 24 23:55:08.288505 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.288477    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.296765 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.296717    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76599bc7bd-jsk4c"]
Apr 24 23:55:08.364536 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-serving-cert\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364536 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-oauth-serving-cert\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364536 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8x5\" (UniqueName: \"kubernetes.io/projected/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-kube-api-access-fz8x5\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364807 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-service-ca\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364807 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-config\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364807 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364659    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff43f34c-be72-4604-b7a1-be1ab3ba9ce4-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-n8z9m\" (UID: \"ff43f34c-be72-4604-b7a1-be1ab3ba9ce4\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:08.364807 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-oauth-config\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364807 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.364709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-trusted-ca-bundle\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.364999 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:08.364829    2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Apr 24 23:55:08.364999 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:08.364894    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff43f34c-be72-4604-b7a1-be1ab3ba9ce4-monitoring-plugin-cert podName:ff43f34c-be72-4604-b7a1-be1ab3ba9ce4 nodeName:}" failed. No retries permitted until 2026-04-24 23:55:08.864876397 +0000 UTC m=+68.489916056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ff43f34c-be72-4604-b7a1-be1ab3ba9ce4-monitoring-plugin-cert") pod "monitoring-plugin-7dccd58f55-n8z9m" (UID: "ff43f34c-be72-4604-b7a1-be1ab3ba9ce4") : secret "monitoring-plugin-cert" not found
Apr 24 23:55:08.374673 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.374559    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5776cf486c-bmk6q"]
Apr 24 23:55:08.378101 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:08.378071    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad93192c_534f_4ab1_ae0d_091859d832b2.slice/crio-3c653f48716b9b4e480d2ed5cf9f4d2ecd2adac0ad917205be82581a7bc569d3 WatchSource:0}: Error finding container 3c653f48716b9b4e480d2ed5cf9f4d2ecd2adac0ad917205be82581a7bc569d3: Status 404 returned error can't find the container with id 3c653f48716b9b4e480d2ed5cf9f4d2ecd2adac0ad917205be82581a7bc569d3
Apr 24 23:55:08.465339 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465304    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-serving-cert\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.465453 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465354    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-oauth-serving-cert\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.465453 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465428    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8x5\" (UniqueName: \"kubernetes.io/projected/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-kube-api-access-fz8x5\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.465559 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465457    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-service-ca\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.465559 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465503    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-config\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.465559 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465552    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-oauth-config\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.465735 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.465577    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-trusted-ca-bundle\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.467068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.466440    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-service-ca\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.467068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.466506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-config\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.467068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.466708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-trusted-ca-bundle\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.467068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.466996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-oauth-serving-cert\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.468591 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.468477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-serving-cert\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.472312 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.472277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-oauth-config\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.478012 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.477983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8x5\" (UniqueName: \"kubernetes.io/projected/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-kube-api-access-fz8x5\") pod \"console-76599bc7bd-jsk4c\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") " pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.607633 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.604589    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:08.607633 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.606786    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"]
Apr 24 23:55:08.611259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.611214    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.618651 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.613827    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Apr 24 23:55:08.618651 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.614064    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Apr 24 23:55:08.618651 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.614225    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Apr 24 23:55:08.618651 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.614403    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zng9s\""
Apr 24 23:55:08.618651 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.614583    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Apr 24 23:55:08.618651 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.615763    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Apr 24 23:55:08.620895 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.620753    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Apr 24 23:55:08.631388 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.631345    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"]
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-secret-telemeter-client\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-federate-client-tls\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-telemeter-client-tls\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knmz9\" (UniqueName: \"kubernetes.io/projected/8cb439d7-f947-477e-a49c-be8a9e3b3c32-kube-api-access-knmz9\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-metrics-client-ca\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.667643 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.667464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-serving-certs-ca-bundle\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768589    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-serving-certs-ca-bundle\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768656    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-secret-telemeter-client\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768678    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-federate-client-tls\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768699    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-telemeter-client-tls\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768738    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768763    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-knmz9\" (UniqueName: \"kubernetes.io/projected/8cb439d7-f947-477e-a49c-be8a9e3b3c32-kube-api-access-knmz9\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768803    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.768826 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.768830    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-metrics-client-ca\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.769916 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.769420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-metrics-client-ca\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.771276 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.770694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-serving-certs-ca-bundle\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.771543 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.771491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb439d7-f947-477e-a49c-be8a9e3b3c32-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.774946 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.774921    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-telemeter-client-tls\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.775631 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.775556    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.776342 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.776299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-federate-client-tls\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.779948 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.779912    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knmz9\" (UniqueName: \"kubernetes.io/projected/8cb439d7-f947-477e-a49c-be8a9e3b3c32-kube-api-access-knmz9\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.780926 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.780907    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8cb439d7-f947-477e-a49c-be8a9e3b3c32-secret-telemeter-client\") pod \"telemeter-client-6fb4d7d57c-ws2z5\" (UID: \"8cb439d7-f947-477e-a49c-be8a9e3b3c32\") " pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:08.797850 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.797488    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76599bc7bd-jsk4c"]
Apr 24 23:55:08.870571 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.870479    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff43f34c-be72-4604-b7a1-be1ab3ba9ce4-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-n8z9m\" (UID: \"ff43f34c-be72-4604-b7a1-be1ab3ba9ce4\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:08.880413 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.880380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff43f34c-be72-4604-b7a1-be1ab3ba9ce4-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-n8z9m\" (UID: \"ff43f34c-be72-4604-b7a1-be1ab3ba9ce4\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:08.932119 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:08.932085    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"
Apr 24 23:55:09.124932 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.124308    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:09.141659 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.141373    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:55:09.144016 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.143581    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:55:09.159022 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.154767    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:55:09.183214 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.182433    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tcvmr" event={"ID":"5b867a9a-b211-492e-a7c9-6081e587a486","Type":"ContainerStarted","Data":"53f1e1adcf6aac0246b7ebc943daf3bea878a092d8c86007a9af32b047debc86"}
Apr 24 23:55:09.183214 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.182479    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tcvmr" event={"ID":"5b867a9a-b211-492e-a7c9-6081e587a486","Type":"ContainerStarted","Data":"76535e1d67d69b11542f058af763d4a77b13d7eb297e2dfb00616d3569141900"}
Apr 24 23:55:09.185783 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.185756    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q" event={"ID":"ad93192c-534f-4ab1-ae0d-091859d832b2","Type":"ContainerStarted","Data":"3c653f48716b9b4e480d2ed5cf9f4d2ecd2adac0ad917205be82581a7bc569d3"}
Apr 24 23:55:09.192402 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.192272    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:55:09.198652 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.198579    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tcvmr" podStartSLOduration=67.083229937 podStartE2EDuration="1m8.198562781s" podCreationTimestamp="2026-04-24 23:54:01 +0000 UTC" firstStartedPulling="2026-04-24 23:55:07.143218889 +0000 UTC m=+66.768258547" lastFinishedPulling="2026-04-24 23:55:08.258551722 +0000 UTC m=+67.883591391" observedRunningTime="2026-04-24 23:55:09.197933266 +0000 UTC m=+68.822972944" watchObservedRunningTime="2026-04-24 23:55:09.198562781 +0000 UTC m=+68.823602457"
Apr 24 23:55:09.677117 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.674857    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 23:55:09.692312 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.692281    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.696203 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.694500    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 23:55:09.696203 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.695362    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 24 23:55:09.696203 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.695912    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 24 23:55:09.696203 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.696063    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 24 23:55:09.696203 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.696102    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 24 23:55:09.696526 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.696375    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 24 23:55:09.696955 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.696775    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 24 23:55:09.696955 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.696828    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 24 23:55:09.697149 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.697015    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 24 23:55:09.697263 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.697247    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rfx9f\""
Apr 24 23:55:09.697462 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.697443    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-avp842ou2ipvj\""
Apr 24 23:55:09.697674 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.697656    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 24 23:55:09.697792 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.697770    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 24 23:55:09.700320 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.700295    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 24 23:55:09.704195 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.704172    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 24 23:55:09.782516 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782516 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782516 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ba6210-0d2c-4431-9f81-cdfdff431c6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782516 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ba6210-0d2c-4431-9f81-cdfdff431c6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-config\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.782854 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.783259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.783259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.783259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.783259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.782991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.783259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.783009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5bb\" (UniqueName: \"kubernetes.io/projected/45ba6210-0d2c-4431-9f81-cdfdff431c6f-kube-api-access-rv5bb\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884173 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884129    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-config\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884181    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884211    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884233    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884258    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884312    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884337    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5bb\" (UniqueName: \"kubernetes.io/projected/45ba6210-0d2c-4431-9f81-cdfdff431c6f-kube-api-access-rv5bb\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884366    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884394    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884424    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ba6210-0d2c-4431-9f81-cdfdff431c6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884446    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884484    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884509    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884531    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884560    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884593    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.884745 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.884706    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ba6210-0d2c-4431-9f81-cdfdff431c6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.885474 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.885405    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.885474 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.885466    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.888315 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.888284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ba6210-0d2c-4431-9f81-cdfdff431c6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.888515 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.888494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.889237 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.888912    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.889237 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.888915    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-config\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.891155 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.890379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.891155 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.890429    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.891155 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.890816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.891155 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.891030    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.891405 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.891230    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.891954 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.891907    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ba6210-0d2c-4431-9f81-cdfdff431c6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.892165 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.892137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.892533 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.892510    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.892667 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.892645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.893441 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.893417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.894429 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.894409    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45ba6210-0d2c-4431-9f81-cdfdff431c6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.894508 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.894488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45ba6210-0d2c-4431-9f81-cdfdff431c6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:09.899869 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:09.899837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5bb\" (UniqueName: \"kubernetes.io/projected/45ba6210-0d2c-4431-9f81-cdfdff431c6f-kube-api-access-rv5bb\") pod \"prometheus-k8s-0\" (UID: \"45ba6210-0d2c-4431-9f81-cdfdff431c6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:10.005134 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:10.005037    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:10.193848 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:10.193810    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76599bc7bd-jsk4c" event={"ID":"dc5530ae-07b6-431d-bd2a-9775dcf14ea2","Type":"ContainerStarted","Data":"ba5e157b614cc9ab3e88460c1cc7f784584c8c289b51597f34e297beba5dc5e6"}
Apr 24 23:55:10.726770 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:10.726719    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5"]
Apr 24 23:55:10.745543 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:10.745508    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"]
Apr 24 23:55:11.609130 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:11.609099    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff43f34c_be72_4604_b7a1_be1ab3ba9ce4.slice/crio-b7020a29b336a7e90d37f8ab808144259ee09e02f65996075697d9b740d548a0 WatchSource:0}: Error finding container b7020a29b336a7e90d37f8ab808144259ee09e02f65996075697d9b740d548a0: Status 404 returned error can't find the container with id b7020a29b336a7e90d37f8ab808144259ee09e02f65996075697d9b740d548a0
Apr 24 23:55:11.610201 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:55:11.610164    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb439d7_f947_477e_a49c_be8a9e3b3c32.slice/crio-f010b0c983e270c800eb62704cc3abf4a1675572471210c962ae16977c8bef3e WatchSource:0}: Error finding container f010b0c983e270c800eb62704cc3abf4a1675572471210c962ae16977c8bef3e: Status 404 returned error can't find the container with id f010b0c983e270c800eb62704cc3abf4a1675572471210c962ae16977c8bef3e
Apr 24 23:55:11.786259 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:11.786194    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 23:55:12.204170 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.204049    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"ce9053b991d698cb6b887a237bca073b34a299f3d4d173092f594bdec6c78419"}
Apr 24 23:55:12.204170 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.204094    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"c6e61d79093feb7b54a1abd56e4abef3734ddc844907629bff50ef236458c090"}
Apr 24 23:55:12.204170 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.204107    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"cc6c8ad5b210f5a343a7bf8903b3a2c6547e7dce7994b4a6f8719abc3e4dd8f3"}
Apr 24 23:55:12.204170 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.204121    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"ad878f86aac9c6426d0778c85a5e911d8fbfc5e35e48bf7adb0bfdbd39c8561c"}
Apr 24 23:55:12.204170 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.204133    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"591b5014bfe5d8ace21f285ff483ea67d36aa525c77a038769dfbfeced7fa032"}
Apr 24 23:55:12.205500 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.205453    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5" event={"ID":"8cb439d7-f947-477e-a49c-be8a9e3b3c32","Type":"ContainerStarted","Data":"f010b0c983e270c800eb62704cc3abf4a1675572471210c962ae16977c8bef3e"}
Apr 24 23:55:12.208096 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.207697    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q" event={"ID":"ad93192c-534f-4ab1-ae0d-091859d832b2","Type":"ContainerStarted","Data":"3950bfbf940a68e12cfb47f65f933d4d1ca407346d35a62112514aa31380d6b0"}
Apr 24 23:55:12.209629 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.209184    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9ltqc" event={"ID":"b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3","Type":"ContainerStarted","Data":"9348ae1c5300e7e7a6553ed810ba15055f7dd9a98ca3820863255bc1296dab16"}
Apr 24 23:55:12.209629 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.209588    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:55:12.212068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.211936    2576 generic.go:358] "Generic (PLEG): container finished" podID="45ba6210-0d2c-4431-9f81-cdfdff431c6f" containerID="d6de17065cf2b82945fd1f711027d02f33b6357aebaa486c2eba4efa5e99e4e5" exitCode=0
Apr 24 23:55:12.212068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.212024    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerDied","Data":"d6de17065cf2b82945fd1f711027d02f33b6357aebaa486c2eba4efa5e99e4e5"}
Apr 24 23:55:12.212068 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.212045    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"5a6aba171d47b5e32482cb6a00fa17f0142d2bed082baa2363b7dfd8b662ca93"}
Apr 24 23:55:12.215477 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.215447    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"10059d18a5f81ea7ba194e7a4d192efb139063a09e22db969afc23dccffddb95"}
Apr 24 23:55:12.215477 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.215475    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"41ebb2647dbc2c360f18742c510bf993e553044357656d073dcebdb1e0d5e43e"}
Apr 24 23:55:12.215652 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.215488    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"4e57e58c7e55df39282a9590943f41e4a168d8b015e378f8d01686baa02b5956"}
Apr 24 23:55:12.217080 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.217057    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76599bc7bd-jsk4c" event={"ID":"dc5530ae-07b6-431d-bd2a-9775dcf14ea2","Type":"ContainerStarted","Data":"7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f"}
Apr 24 23:55:12.218346 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.218310    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m" event={"ID":"ff43f34c-be72-4604-b7a1-be1ab3ba9ce4","Type":"ContainerStarted","Data":"b7020a29b336a7e90d37f8ab808144259ee09e02f65996075697d9b740d548a0"}
Apr 24 23:55:12.224639 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.224325    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q" podStartSLOduration=1.982380927 podStartE2EDuration="5.224310097s" podCreationTimestamp="2026-04-24 23:55:07 +0000 UTC" firstStartedPulling="2026-04-24 23:55:08.380936241 +0000 UTC m=+68.005975895" lastFinishedPulling="2026-04-24 23:55:11.622865411 +0000 UTC m=+71.247905065" observedRunningTime="2026-04-24 23:55:12.222840502 +0000 UTC m=+71.847880179" watchObservedRunningTime="2026-04-24 23:55:12.224310097 +0000 UTC m=+71.849349774"
Apr 24 23:55:12.240338 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.240283    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-76599bc7bd-jsk4c" podStartSLOduration=4.240267367 podStartE2EDuration="4.240267367s" podCreationTimestamp="2026-04-24 23:55:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:55:12.237336198 +0000 UTC m=+71.862375891" watchObservedRunningTime="2026-04-24 23:55:12.240267367 +0000 UTC m=+71.865307044"
Apr 24 23:55:12.258075 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:12.257670    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9ltqc" podStartSLOduration=66.956528913 podStartE2EDuration="1m11.257652828s" podCreationTimestamp="2026-04-24 23:54:01 +0000 UTC" firstStartedPulling="2026-04-24 23:55:07.368736409 +0000 UTC m=+66.993776064" lastFinishedPulling="2026-04-24 23:55:11.669860308 +0000 UTC m=+71.294899979" observedRunningTime="2026-04-24 23:55:12.255980523 +0000 UTC m=+71.881020202" watchObservedRunningTime="2026-04-24 23:55:12.257652828 +0000 UTC m=+71.882692506"
Apr 24 23:55:14.229537 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.229495    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"0bf0b0ddec3c26d561b0c30b980e16e7e8f1dada6e9b686f80cdd2f3a9e720db"}
Apr 24 23:55:14.229537 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.229542    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"4fb67f78d060bcd7cf6ef4a6718887b0ffaa8fb8e84f523fa385e10d60b47ef5"}
Apr 24 23:55:14.231464 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.231432    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m" event={"ID":"ff43f34c-be72-4604-b7a1-be1ab3ba9ce4","Type":"ContainerStarted","Data":"f1013d5b2f85b643b2d015844dabee619cf79edc092fbe2c7723e6c2fa9ab85c"}
Apr 24 23:55:14.231659 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.231641    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:14.234948 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.234919    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54","Type":"ContainerStarted","Data":"a57b0f23db7004d963f99f94c108c2288f99cb81a11757e2d0bc6d198dff2ba5"}
Apr 24 23:55:14.237758 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.237732    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5" event={"ID":"8cb439d7-f947-477e-a49c-be8a9e3b3c32","Type":"ContainerStarted","Data":"7b81c92bcf636f7218d784a3d5e52493ae001fb1b0e2902cf0237d659c3d0a42"}
Apr 24 23:55:14.237900 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.237762    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5" event={"ID":"8cb439d7-f947-477e-a49c-be8a9e3b3c32","Type":"ContainerStarted","Data":"ffe6a31c75dd57fae625a439997265999555c3d21166c948f5e7f2901cec541d"}
Apr 24 23:55:14.237989 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.237971    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m"
Apr 24 23:55:14.247149 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.247109    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-n8z9m" podStartSLOduration=4.077333876 podStartE2EDuration="6.247096371s" podCreationTimestamp="2026-04-24 23:55:08 +0000 UTC" firstStartedPulling="2026-04-24 23:55:11.620742864 +0000 UTC m=+71.245782522" lastFinishedPulling="2026-04-24 23:55:13.790505349 +0000 UTC m=+73.415545017" observedRunningTime="2026-04-24 23:55:14.245704264 +0000 UTC m=+73.870743961" watchObservedRunningTime="2026-04-24 23:55:14.247096371 +0000 UTC m=+73.872136046"
Apr 24 23:55:14.271061 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:14.271013    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.340954962 podStartE2EDuration="10.270998899s" podCreationTimestamp="2026-04-24 23:55:04 +0000 UTC" firstStartedPulling="2026-04-24 23:55:05.860695098 +0000 UTC m=+65.485734766" lastFinishedPulling="2026-04-24 23:55:13.790739046 +0000 UTC m=+73.415778703" observedRunningTime="2026-04-24 23:55:14.269314952 +0000 UTC m=+73.894354639" watchObservedRunningTime="2026-04-24 23:55:14.270998899 +0000 UTC m=+73.896038574"
Apr 24 23:55:15.246042 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.246003    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5" event={"ID":"8cb439d7-f947-477e-a49c-be8a9e3b3c32","Type":"ContainerStarted","Data":"1215324dde65b9e14c8793ed14064e7b2f56b169631d412b2683c6293cd5d68b"}
Apr 24 23:55:15.254037 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.253986    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"19e75accea88a15917573aa4c498cb6a9be8f9d0e191c96792a6128b8a57c7dc"}
Apr 24 23:55:15.254037 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.254030    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"7b98d3eefbde8c11a2c57f6de43b3377c5c43b2104dd1d7cc6de18f5eef324af"}
Apr 24 23:55:15.259041 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.258852    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" event={"ID":"6e9a1e5d-0ba8-41db-bb50-31e22229e9b8","Type":"ContainerStarted","Data":"61cade16c0d35b9340d4d8a387822760b679e3342620245c0f35c5ccab4ae97b"}
Apr 24 23:55:15.259041 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.258891    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:15.274074 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.273202    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6fb4d7d57c-ws2z5" podStartSLOduration=5.097890886 podStartE2EDuration="7.273181481s" podCreationTimestamp="2026-04-24 23:55:08 +0000 UTC" firstStartedPulling="2026-04-24 23:55:11.620793292 +0000 UTC m=+71.245832960" lastFinishedPulling="2026-04-24 23:55:13.796083898 +0000 UTC m=+73.421123555" observedRunningTime="2026-04-24 23:55:15.272209497 +0000 UTC m=+74.897249217" watchObservedRunningTime="2026-04-24 23:55:15.273181481 +0000 UTC m=+74.898221158"
Apr 24 23:55:15.297303 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:15.297177    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q" podStartSLOduration=2.883008492 podStartE2EDuration="9.297154641s" podCreationTimestamp="2026-04-24 23:55:06 +0000 UTC" firstStartedPulling="2026-04-24 23:55:07.375982385 +0000 UTC m=+67.001022043" lastFinishedPulling="2026-04-24 23:55:13.79012852 +0000 UTC m=+73.415168192" observedRunningTime="2026-04-24 23:55:15.295585416 +0000 UTC m=+74.920625093" watchObservedRunningTime="2026-04-24 23:55:15.297154641 +0000 UTC m=+74.922194319"
Apr 24 23:55:16.020903 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.020853    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76599bc7bd-jsk4c"]
Apr 24 23:55:16.265548 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.265508    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"15f4b586f88e7983b392be0089fd894418202f06cc2540cb9c6070b42e8232d5"}
Apr 24 23:55:16.265548 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.265551    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"d6e10bf1fcb352ac809380af82b2ebbff0ab3f02261e0f7a2bb6b4ee7e361608"}
Apr 24 23:55:16.266023 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.265563    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"d2ccb160c7484f2ed0a8b92acf7955c1ac9f370b8715c2d42695662e0999d82a"}
Apr 24 23:55:16.266023 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.265571    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45ba6210-0d2c-4431-9f81-cdfdff431c6f","Type":"ContainerStarted","Data":"4db918b1d1edd14059a84e3d4d4a489c703b7ec613c7586f885a566b1eadc766"}
Apr 24 23:55:16.272363 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.272288    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dbb5b4cc9-d9h9q"
Apr 24 23:55:16.297357 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:16.297303    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.481305833 podStartE2EDuration="7.297284908s" podCreationTimestamp="2026-04-24 23:55:09 +0000 UTC" firstStartedPulling="2026-04-24 23:55:12.213385987 +0000 UTC m=+71.838425644" lastFinishedPulling="2026-04-24 23:55:15.029365047 +0000 UTC m=+74.654404719" observedRunningTime="2026-04-24 23:55:16.29471216 +0000 UTC m=+75.919751862" watchObservedRunningTime="2026-04-24 23:55:16.297284908 +0000 UTC m=+75.922324585"
Apr 24 23:55:18.605568 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:18.605528    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:20.006034 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:20.006001    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:55:26.465320 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:26.465281    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b8f44ddb-9fh26"]
Apr 24 23:55:28.126393 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:28.126357    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:28.126393 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:28.126397    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:41.044695 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.044651    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-76599bc7bd-jsk4c" podUID="dc5530ae-07b6-431d-bd2a-9775dcf14ea2" containerName="console" containerID="cri-o://7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f" gracePeriod=15
Apr 24 23:55:41.311167 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.311143    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76599bc7bd-jsk4c_dc5530ae-07b6-431d-bd2a-9775dcf14ea2/console/0.log"
Apr 24 23:55:41.311288 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.311220    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:41.342030 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.342005    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76599bc7bd-jsk4c_dc5530ae-07b6-431d-bd2a-9775dcf14ea2/console/0.log"
Apr 24 23:55:41.342197 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.342048    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc5530ae-07b6-431d-bd2a-9775dcf14ea2" containerID="7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f" exitCode=2
Apr 24 23:55:41.342197 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.342092    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76599bc7bd-jsk4c" event={"ID":"dc5530ae-07b6-431d-bd2a-9775dcf14ea2","Type":"ContainerDied","Data":"7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f"}
Apr 24 23:55:41.342197 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.342114    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76599bc7bd-jsk4c"
Apr 24 23:55:41.342197 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.342132    2576 scope.go:117] "RemoveContainer" containerID="7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f"
Apr 24 23:55:41.342378 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.342118    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76599bc7bd-jsk4c" event={"ID":"dc5530ae-07b6-431d-bd2a-9775dcf14ea2","Type":"ContainerDied","Data":"ba5e157b614cc9ab3e88460c1cc7f784584c8c289b51597f34e297beba5dc5e6"}
Apr 24 23:55:41.371017 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.370995    2576 scope.go:117] "RemoveContainer" containerID="7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f"
Apr 24 23:55:41.371360 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:41.371343    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f\": container with ID starting with 7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f not found: ID does not exist" containerID="7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f"
Apr 24 23:55:41.371409 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.371372    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f"} err="failed to get container status \"7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f\": rpc error: code = NotFound desc = could not find container \"7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f\": container with ID starting with 7b3a9db01066d51ef0d22dd7f5a4b022efd2f2ababd9403b0cc4104c0c9f209f not found: ID does not exist"
Apr 24 23:55:41.388657 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388631    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-service-ca\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.388754 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388664    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-config\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.388754 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388704    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-oauth-serving-cert\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.388754 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388728    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-trusted-ca-bundle\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.388901 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388835    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-serving-cert\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.388959 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388907    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:41.388959 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388919    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz8x5\" (UniqueName: \"kubernetes.io/projected/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-kube-api-access-fz8x5\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.388959 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.388945    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-oauth-config\") pod \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\" (UID: \"dc5530ae-07b6-431d-bd2a-9775dcf14ea2\") "
Apr 24 23:55:41.389737 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.389508    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:41.389737 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.389508    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-config" (OuterVolumeSpecName: "console-config") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:41.389737 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.389566    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:41.391764 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.390222    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-service-ca\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.391764 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.390251    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-config\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.391764 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.390266    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-oauth-serving-cert\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.391764 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.390289    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-trusted-ca-bundle\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.392006 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.391964    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:55:41.394935 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.393704    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:55:41.395496 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.395476    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-kube-api-access-fz8x5" (OuterVolumeSpecName: "kube-api-access-fz8x5") pod "dc5530ae-07b6-431d-bd2a-9775dcf14ea2" (UID: "dc5530ae-07b6-431d-bd2a-9775dcf14ea2"). InnerVolumeSpecName "kube-api-access-fz8x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:55:41.491036 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.490990    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-serving-cert\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.491036 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.491030    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8x5\" (UniqueName: \"kubernetes.io/projected/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-kube-api-access-fz8x5\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.491036 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.491044    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc5530ae-07b6-431d-bd2a-9775dcf14ea2-console-oauth-config\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:41.662827 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.662789    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76599bc7bd-jsk4c"]
Apr 24 23:55:41.669222 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:41.669194    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76599bc7bd-jsk4c"]
Apr 24 23:55:42.873843 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:42.873799    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc5530ae-07b6-431d-bd2a-9775dcf14ea2" path="/var/lib/kubelet/pods/dc5530ae-07b6-431d-bd2a-9775dcf14ea2/volumes"
Apr 24 23:55:44.240490 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:44.240465    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9ltqc"
Apr 24 23:55:48.131422 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:48.131387    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:48.135376 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:48.135353    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5776cf486c-bmk6q"
Apr 24 23:55:51.486244 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.486177    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-68b8f44ddb-9fh26" podUID="1614041e-bf4b-4117-92ef-bfdd7a9543f7" containerName="console" containerID="cri-o://ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1" gracePeriod=15
Apr 24 23:55:51.769510 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.769485    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68b8f44ddb-9fh26_1614041e-bf4b-4117-92ef-bfdd7a9543f7/console/0.log"
Apr 24 23:55:51.769711 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.769555    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:55:51.903373 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903325    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-oauth-config\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.903573 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903402    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-service-ca\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.903573 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903455    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-config\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.903573 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903489    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-serving-cert\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.903573 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903551    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-oauth-serving-cert\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.903867 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903619    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-trusted-ca-bundle\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.903867 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.903674    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhg8h\" (UniqueName: \"kubernetes.io/projected/1614041e-bf4b-4117-92ef-bfdd7a9543f7-kube-api-access-xhg8h\") pod \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\" (UID: \"1614041e-bf4b-4117-92ef-bfdd7a9543f7\") "
Apr 24 23:55:51.906668 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.905912    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:51.906668 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.906195    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:51.906668 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.906452    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:51.908179 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.907031    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-config" (OuterVolumeSpecName: "console-config") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:55:51.908179 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.907138    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:55:51.910561 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.910520    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:55:51.911338 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:51.911310    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1614041e-bf4b-4117-92ef-bfdd7a9543f7-kube-api-access-xhg8h" (OuterVolumeSpecName: "kube-api-access-xhg8h") pod "1614041e-bf4b-4117-92ef-bfdd7a9543f7" (UID: "1614041e-bf4b-4117-92ef-bfdd7a9543f7"). InnerVolumeSpecName "kube-api-access-xhg8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:55:52.005812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005730    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-oauth-serving-cert\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.005812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005767    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-trusted-ca-bundle\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.005812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005782    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhg8h\" (UniqueName: \"kubernetes.io/projected/1614041e-bf4b-4117-92ef-bfdd7a9543f7-kube-api-access-xhg8h\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.005812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005794    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-oauth-config\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.005812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005805    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-service-ca\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.005812 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005813    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-config\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.006122 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.005822    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1614041e-bf4b-4117-92ef-bfdd7a9543f7-console-serving-cert\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 24 23:55:52.382638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.381198    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68b8f44ddb-9fh26_1614041e-bf4b-4117-92ef-bfdd7a9543f7/console/0.log"
Apr 24 23:55:52.382638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.381242    2576 generic.go:358] "Generic (PLEG): container finished" podID="1614041e-bf4b-4117-92ef-bfdd7a9543f7" containerID="ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1" exitCode=2
Apr 24 23:55:52.382638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.381376    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b8f44ddb-9fh26" event={"ID":"1614041e-bf4b-4117-92ef-bfdd7a9543f7","Type":"ContainerDied","Data":"ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1"}
Apr 24 23:55:52.382638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.381408    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b8f44ddb-9fh26" event={"ID":"1614041e-bf4b-4117-92ef-bfdd7a9543f7","Type":"ContainerDied","Data":"6401604a315ae12c64dd8dec892051e9f893a19e549509978b62f80873631a4b"}
Apr 24 23:55:52.382638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.381429    2576 scope.go:117] "RemoveContainer" containerID="ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1"
Apr 24 23:55:52.382638 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.381585    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b8f44ddb-9fh26"
Apr 24 23:55:52.401696 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.401670    2576 scope.go:117] "RemoveContainer" containerID="ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1"
Apr 24 23:55:52.401961 ip-10-0-134-82 kubenswrapper[2576]: E0424 23:55:52.401940    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1\": container with ID starting with ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1 not found: ID does not exist" containerID="ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1"
Apr 24 23:55:52.402032 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.401968    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1"} err="failed to get container status \"ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1\": rpc error: code = NotFound desc = could not find container \"ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1\": container with ID starting with ee300ae92f10b5f515c12cd64ddf76370fb21c05ff4a968e39eff5d8d0cbd3e1 not found: ID does not exist"
Apr 24 23:55:52.412035 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.412008    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b8f44ddb-9fh26"]
Apr 24 23:55:52.417047 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.417022    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68b8f44ddb-9fh26"]
Apr 24 23:55:52.876064 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:55:52.876032    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1614041e-bf4b-4117-92ef-bfdd7a9543f7" path="/var/lib/kubelet/pods/1614041e-bf4b-4117-92ef-bfdd7a9543f7/volumes"
Apr 24 23:56:10.005310 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:10.005267    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:56:10.024336 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:10.024311    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:56:10.449299 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:10.449273    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 23:56:54.387255 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387176    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9m6vr"]
Apr 24 23:56:54.387730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387546    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1614041e-bf4b-4117-92ef-bfdd7a9543f7" containerName="console"
Apr 24 23:56:54.387730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387558    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1614041e-bf4b-4117-92ef-bfdd7a9543f7" containerName="console"
Apr 24 23:56:54.387730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387567    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc5530ae-07b6-431d-bd2a-9775dcf14ea2" containerName="console"
Apr 24 23:56:54.387730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387572    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc5530ae-07b6-431d-bd2a-9775dcf14ea2" containerName="console"
Apr 24 23:56:54.387730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387653    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1614041e-bf4b-4117-92ef-bfdd7a9543f7" containerName="console"
Apr 24 23:56:54.387730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.387665    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc5530ae-07b6-431d-bd2a-9775dcf14ea2" containerName="console"
Apr 24 23:56:54.390777 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.390760    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.393097 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.393061    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 24 23:56:54.397066 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.397041    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9m6vr"]
Apr 24 23:56:54.552829 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.552794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/762b6285-4d40-434c-b48c-cc06271acb74-original-pull-secret\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.553004 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.552852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/762b6285-4d40-434c-b48c-cc06271acb74-dbus\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.553004 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.552888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/762b6285-4d40-434c-b48c-cc06271acb74-kubelet-config\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.653453 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.653363    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/762b6285-4d40-434c-b48c-cc06271acb74-kubelet-config\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.653453 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.653445    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/762b6285-4d40-434c-b48c-cc06271acb74-original-pull-secret\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.653664 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.653467    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/762b6285-4d40-434c-b48c-cc06271acb74-dbus\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.653664 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.653495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/762b6285-4d40-434c-b48c-cc06271acb74-kubelet-config\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.653664 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.653587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/762b6285-4d40-434c-b48c-cc06271acb74-dbus\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.655964 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.655944    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/762b6285-4d40-434c-b48c-cc06271acb74-original-pull-secret\") pod \"global-pull-secret-syncer-9m6vr\" (UID: \"762b6285-4d40-434c-b48c-cc06271acb74\") " pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.700829 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.700793    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9m6vr"
Apr 24 23:56:54.820427 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:54.820305    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9m6vr"]
Apr 24 23:56:54.823077 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:56:54.823040    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762b6285_4d40_434c_b48c_cc06271acb74.slice/crio-846bca572d285b2fa274364335171c290f6670239089208c2914410f81e81db3 WatchSource:0}: Error finding container 846bca572d285b2fa274364335171c290f6670239089208c2914410f81e81db3: Status 404 returned error can't find the container with id 846bca572d285b2fa274364335171c290f6670239089208c2914410f81e81db3
Apr 24 23:56:55.578310 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:55.578272    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9m6vr" event={"ID":"762b6285-4d40-434c-b48c-cc06271acb74","Type":"ContainerStarted","Data":"846bca572d285b2fa274364335171c290f6670239089208c2914410f81e81db3"}
Apr 24 23:56:58.589723 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:58.589683    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9m6vr" event={"ID":"762b6285-4d40-434c-b48c-cc06271acb74","Type":"ContainerStarted","Data":"c10881e3cce1e968d6481e2e5d3a1d49a09d080d47db20a336dff7353a9bf95d"}
Apr 24 23:56:58.604675 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:56:58.604594    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9m6vr" podStartSLOduration=1.217507446 podStartE2EDuration="4.604579151s" podCreationTimestamp="2026-04-24 23:56:54 +0000 UTC" firstStartedPulling="2026-04-24 23:56:54.82492538 +0000 UTC m=+174.449965037" lastFinishedPulling="2026-04-24 23:56:58.211997073 +0000 UTC m=+177.837036742" observedRunningTime="2026-04-24 23:56:58.603740343 +0000 UTC m=+178.228780020" watchObservedRunningTime="2026-04-24 23:56:58.604579151 +0000 UTC m=+178.229618827"
Apr 24 23:57:41.371330 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.371287    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-zgnpg"]
Apr 24 23:57:41.375545 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.375519    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.378045 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.378016    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Apr 24 23:57:41.378186 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.378060    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Apr 24 23:57:41.378818 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.378800    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5rxdj\""
Apr 24 23:57:41.378906 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.378804    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Apr 24 23:57:41.379079 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.379063    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Apr 24 23:57:41.382616 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.382578    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-zgnpg"]
Apr 24 23:57:41.458089 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.458054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7sk\" (UniqueName: \"kubernetes.io/projected/691335ae-537e-4f17-b6e7-623981b5b707-kube-api-access-wg7sk\") pod \"keda-admission-cf49989db-zgnpg\" (UID: \"691335ae-537e-4f17-b6e7-623981b5b707\") " pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.458269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.458114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/691335ae-537e-4f17-b6e7-623981b5b707-certificates\") pod \"keda-admission-cf49989db-zgnpg\" (UID: \"691335ae-537e-4f17-b6e7-623981b5b707\") " pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.559487 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.559444    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7sk\" (UniqueName: \"kubernetes.io/projected/691335ae-537e-4f17-b6e7-623981b5b707-kube-api-access-wg7sk\") pod \"keda-admission-cf49989db-zgnpg\" (UID: \"691335ae-537e-4f17-b6e7-623981b5b707\") " pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.559716 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.559638    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/691335ae-537e-4f17-b6e7-623981b5b707-certificates\") pod \"keda-admission-cf49989db-zgnpg\" (UID: \"691335ae-537e-4f17-b6e7-623981b5b707\") " pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.562498 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.562469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/691335ae-537e-4f17-b6e7-623981b5b707-certificates\") pod \"keda-admission-cf49989db-zgnpg\" (UID: \"691335ae-537e-4f17-b6e7-623981b5b707\") " pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.567751 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.567721    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7sk\" (UniqueName: \"kubernetes.io/projected/691335ae-537e-4f17-b6e7-623981b5b707-kube-api-access-wg7sk\") pod \"keda-admission-cf49989db-zgnpg\" (UID: \"691335ae-537e-4f17-b6e7-623981b5b707\") " pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.687628 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.687503    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:41.835187 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:41.835082    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-zgnpg"]
Apr 24 23:57:41.837470 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:57:41.837435    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691335ae_537e_4f17_b6e7_623981b5b707.slice/crio-30850c1a647c031cb928f81fb04d5adae8a788cd089a1e925f9b5efee8e2154b WatchSource:0}: Error finding container 30850c1a647c031cb928f81fb04d5adae8a788cd089a1e925f9b5efee8e2154b: Status 404 returned error can't find the container with id 30850c1a647c031cb928f81fb04d5adae8a788cd089a1e925f9b5efee8e2154b
Apr 24 23:57:42.740711 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:42.740668    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-zgnpg" event={"ID":"691335ae-537e-4f17-b6e7-623981b5b707","Type":"ContainerStarted","Data":"30850c1a647c031cb928f81fb04d5adae8a788cd089a1e925f9b5efee8e2154b"}
Apr 24 23:57:46.754753 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:46.754718    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-zgnpg" event={"ID":"691335ae-537e-4f17-b6e7-623981b5b707","Type":"ContainerStarted","Data":"e4e4ae4dee942e86bfdaf62a2afdc42b416f5ef777a889822d07258cbea66dde"}
Apr 24 23:57:46.755172 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:46.754874    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:57:46.772755 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:57:46.772706    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-zgnpg" podStartSLOduration=1.832337811 podStartE2EDuration="5.772690259s" podCreationTimestamp="2026-04-24 23:57:41 +0000 UTC" firstStartedPulling="2026-04-24 23:57:41.838738435 +0000 UTC m=+221.463778089" lastFinishedPulling="2026-04-24 23:57:45.779090883 +0000 UTC m=+225.404130537" observedRunningTime="2026-04-24 23:57:46.770763691 +0000 UTC m=+226.395803369" watchObservedRunningTime="2026-04-24 23:57:46.772690259 +0000 UTC m=+226.397729934"
Apr 24 23:58:07.766709 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:07.766675    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-zgnpg"
Apr 24 23:58:52.085356 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.085311    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-mrwx8"]
Apr 24 23:58:52.088704 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.088682    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.091337 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.091317    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Apr 24 23:58:52.091833 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.091815    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Apr 24 23:58:52.092317 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.092298    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-72h85\""
Apr 24 23:58:52.097457 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.097435    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-mrwx8"]
Apr 24 23:58:52.164940 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.164885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5618f446-82d8-4908-8f12-9b644b60c8fd-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-mrwx8\" (UID: \"5618f446-82d8-4908-8f12-9b644b60c8fd\") " pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.165121 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.165003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxwx\" (UniqueName: \"kubernetes.io/projected/5618f446-82d8-4908-8f12-9b644b60c8fd-kube-api-access-7pxwx\") pod \"cert-manager-webhook-587ccfb98-mrwx8\" (UID: \"5618f446-82d8-4908-8f12-9b644b60c8fd\") " pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.266319 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.266260    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxwx\" (UniqueName: \"kubernetes.io/projected/5618f446-82d8-4908-8f12-9b644b60c8fd-kube-api-access-7pxwx\") pod \"cert-manager-webhook-587ccfb98-mrwx8\" (UID: \"5618f446-82d8-4908-8f12-9b644b60c8fd\") " pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.266503 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.266365    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5618f446-82d8-4908-8f12-9b644b60c8fd-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-mrwx8\" (UID: \"5618f446-82d8-4908-8f12-9b644b60c8fd\") " pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.274225 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.274190    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5618f446-82d8-4908-8f12-9b644b60c8fd-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-mrwx8\" (UID: \"5618f446-82d8-4908-8f12-9b644b60c8fd\") " pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.274352 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.274307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxwx\" (UniqueName: \"kubernetes.io/projected/5618f446-82d8-4908-8f12-9b644b60c8fd-kube-api-access-7pxwx\") pod \"cert-manager-webhook-587ccfb98-mrwx8\" (UID: \"5618f446-82d8-4908-8f12-9b644b60c8fd\") " pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.408124 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.408024    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:58:52.532324 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.532287    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-mrwx8"]
Apr 24 23:58:52.535016 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:58:52.534978    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5618f446_82d8_4908_8f12_9b644b60c8fd.slice/crio-9a3972c13ca93fc6f55498891a90ccefe30d7066f66f44c96946193a60136411 WatchSource:0}: Error finding container 9a3972c13ca93fc6f55498891a90ccefe30d7066f66f44c96946193a60136411: Status 404 returned error can't find the container with id 9a3972c13ca93fc6f55498891a90ccefe30d7066f66f44c96946193a60136411
Apr 24 23:58:52.944867 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:52.944833    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8" event={"ID":"5618f446-82d8-4908-8f12-9b644b60c8fd","Type":"ContainerStarted","Data":"9a3972c13ca93fc6f55498891a90ccefe30d7066f66f44c96946193a60136411"}
Apr 24 23:58:53.972998 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:53.972964    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-26wg5"]
Apr 24 23:58:53.976082 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:53.976066    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:53.978091 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:53.978074    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qqhs9\""
Apr 24 23:58:53.984751 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:53.984731    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-26wg5"]
Apr 24 23:58:54.083355 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.083317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqsl\" (UniqueName: \"kubernetes.io/projected/bf07815d-d1b6-4350-a890-41ce97d39403-kube-api-access-2vqsl\") pod \"cert-manager-cainjector-68b757865b-26wg5\" (UID: \"bf07815d-d1b6-4350-a890-41ce97d39403\") " pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.083548 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.083438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf07815d-d1b6-4350-a890-41ce97d39403-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-26wg5\" (UID: \"bf07815d-d1b6-4350-a890-41ce97d39403\") " pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.184171 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.184131    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf07815d-d1b6-4350-a890-41ce97d39403-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-26wg5\" (UID: \"bf07815d-d1b6-4350-a890-41ce97d39403\") " pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.184341 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.184204    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqsl\" (UniqueName: \"kubernetes.io/projected/bf07815d-d1b6-4350-a890-41ce97d39403-kube-api-access-2vqsl\") pod \"cert-manager-cainjector-68b757865b-26wg5\" (UID: \"bf07815d-d1b6-4350-a890-41ce97d39403\") " pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.193519 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.193492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqsl\" (UniqueName: \"kubernetes.io/projected/bf07815d-d1b6-4350-a890-41ce97d39403-kube-api-access-2vqsl\") pod \"cert-manager-cainjector-68b757865b-26wg5\" (UID: \"bf07815d-d1b6-4350-a890-41ce97d39403\") " pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.193626 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.193551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf07815d-d1b6-4350-a890-41ce97d39403-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-26wg5\" (UID: \"bf07815d-d1b6-4350-a890-41ce97d39403\") " pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.285878 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.285802    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5"
Apr 24 23:58:54.406416 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.406371    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-26wg5"]
Apr 24 23:58:54.410882 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:58:54.410849    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf07815d_d1b6_4350_a890_41ce97d39403.slice/crio-6e2115a398dd6ba97eb703dc86a9cc770ba6b368c2aa47561d3ee0831607709b WatchSource:0}: Error finding container 6e2115a398dd6ba97eb703dc86a9cc770ba6b368c2aa47561d3ee0831607709b: Status 404 returned error can't find the container with id 6e2115a398dd6ba97eb703dc86a9cc770ba6b368c2aa47561d3ee0831607709b
Apr 24 23:58:54.951669 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:54.951631    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5" event={"ID":"bf07815d-d1b6-4350-a890-41ce97d39403","Type":"ContainerStarted","Data":"6e2115a398dd6ba97eb703dc86a9cc770ba6b368c2aa47561d3ee0831607709b"}
Apr 24 23:58:58.970403 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:58.970360    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5" event={"ID":"bf07815d-d1b6-4350-a890-41ce97d39403","Type":"ContainerStarted","Data":"dce73c564b619b011d7c5c9a31203359e42bc3dc544f34dfb4ad119992a76993"}
Apr 24 23:58:58.985730 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:58.985671    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-26wg5" podStartSLOduration=1.652459436 podStartE2EDuration="5.985655462s" podCreationTimestamp="2026-04-24 23:58:53 +0000 UTC" firstStartedPulling="2026-04-24 23:58:54.41283721 +0000 UTC m=+294.037876864" lastFinishedPulling="2026-04-24 23:58:58.746033222 +0000 UTC m=+298.371072890" observedRunningTime="2026-04-24 23:58:58.984776784 +0000 UTC m=+298.609816460" watchObservedRunningTime="2026-04-24 23:58:58.985655462 +0000 UTC m=+298.610695138"
Apr 24 23:58:59.975143 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:59.975104    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8" event={"ID":"5618f446-82d8-4908-8f12-9b644b60c8fd","Type":"ContainerStarted","Data":"64698bf1d58341789b13b02ba04bad6ab8a923822a3fff5ac78f646480a34d9b"}
Apr 24 23:58:59.975622 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:58:59.975288    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:59:00.001323 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:00.001274    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8" podStartSLOduration=1.545910557 podStartE2EDuration="8.001257937s" podCreationTimestamp="2026-04-24 23:58:52 +0000 UTC" firstStartedPulling="2026-04-24 23:58:52.536785642 +0000 UTC m=+292.161825296" lastFinishedPulling="2026-04-24 23:58:58.992133002 +0000 UTC m=+298.617172676" observedRunningTime="2026-04-24 23:58:59.9999011 +0000 UTC m=+299.624940817" watchObservedRunningTime="2026-04-24 23:59:00.001257937 +0000 UTC m=+299.626297613"
Apr 24 23:59:00.756681 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:00.756650    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 24 23:59:00.756886 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:00.756872    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 24 23:59:00.769077 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:00.769043    2576 kubelet.go:1628] "Image garbage collection succeeded"
Apr 24 23:59:05.980344 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:05.980307    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-mrwx8"
Apr 24 23:59:12.308457 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.308419    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-zn2b5"]
Apr 24 23:59:12.311835 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.311819    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.313974 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.313952    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-p7x2w\""
Apr 24 23:59:12.320689 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.320664    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-zn2b5"]
Apr 24 23:59:12.443591 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.443556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7rp\" (UniqueName: \"kubernetes.io/projected/3ebe6bec-bc38-43fa-a426-b64218ae8f4f-kube-api-access-bs7rp\") pod \"cert-manager-79c8d999ff-zn2b5\" (UID: \"3ebe6bec-bc38-43fa-a426-b64218ae8f4f\") " pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.443809 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.443688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ebe6bec-bc38-43fa-a426-b64218ae8f4f-bound-sa-token\") pod \"cert-manager-79c8d999ff-zn2b5\" (UID: \"3ebe6bec-bc38-43fa-a426-b64218ae8f4f\") " pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.544212 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.544170    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ebe6bec-bc38-43fa-a426-b64218ae8f4f-bound-sa-token\") pod \"cert-manager-79c8d999ff-zn2b5\" (UID: \"3ebe6bec-bc38-43fa-a426-b64218ae8f4f\") " pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.544397 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.544247    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7rp\" (UniqueName: \"kubernetes.io/projected/3ebe6bec-bc38-43fa-a426-b64218ae8f4f-kube-api-access-bs7rp\") pod \"cert-manager-79c8d999ff-zn2b5\" (UID: \"3ebe6bec-bc38-43fa-a426-b64218ae8f4f\") " pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.555843 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.555808    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7rp\" (UniqueName: \"kubernetes.io/projected/3ebe6bec-bc38-43fa-a426-b64218ae8f4f-kube-api-access-bs7rp\") pod \"cert-manager-79c8d999ff-zn2b5\" (UID: \"3ebe6bec-bc38-43fa-a426-b64218ae8f4f\") " pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.556003 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.555980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ebe6bec-bc38-43fa-a426-b64218ae8f4f-bound-sa-token\") pod \"cert-manager-79c8d999ff-zn2b5\" (UID: \"3ebe6bec-bc38-43fa-a426-b64218ae8f4f\") " pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.622075 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.622041    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-zn2b5"
Apr 24 23:59:12.745129 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.745083    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-zn2b5"]
Apr 24 23:59:12.748050 ip-10-0-134-82 kubenswrapper[2576]: W0424 23:59:12.748025    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ebe6bec_bc38_43fa_a426_b64218ae8f4f.slice/crio-ef1d5968c7e4ddfa4cb938b29a66fdd9816632b1139c6bb4109b36aba6826044 WatchSource:0}: Error finding container ef1d5968c7e4ddfa4cb938b29a66fdd9816632b1139c6bb4109b36aba6826044: Status 404 returned error can't find the container with id ef1d5968c7e4ddfa4cb938b29a66fdd9816632b1139c6bb4109b36aba6826044
Apr 24 23:59:12.750253 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:12.750239    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:59:13.012268 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:13.012181    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-zn2b5" event={"ID":"3ebe6bec-bc38-43fa-a426-b64218ae8f4f","Type":"ContainerStarted","Data":"2a828679dc1ed1340eb84711db60db51d5eeee96eff987aa320ccb56919f0fb4"}
Apr 24 23:59:13.012268 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:13.012223    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-zn2b5" event={"ID":"3ebe6bec-bc38-43fa-a426-b64218ae8f4f","Type":"ContainerStarted","Data":"ef1d5968c7e4ddfa4cb938b29a66fdd9816632b1139c6bb4109b36aba6826044"}
Apr 24 23:59:13.030351 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:13.030295    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-zn2b5" podStartSLOduration=1.0302769 podStartE2EDuration="1.0302769s" podCreationTimestamp="2026-04-24 23:59:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:59:13.029010671 +0000 UTC m=+312.654050365" watchObservedRunningTime="2026-04-24 23:59:13.0302769 +0000 UTC m=+312.655316577"
Apr 24 23:59:41.999203 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:41.999165    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"]
Apr 24 23:59:42.005739 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.005690    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.009883 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.009857    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fb2ft\""
Apr 24 23:59:42.010024 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.009861    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Apr 24 23:59:42.010024 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.009934    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Apr 24 23:59:42.010024 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.009920    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Apr 24 23:59:42.010024 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.009936    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Apr 24 23:59:42.010024 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.009879    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Apr 24 23:59:42.018061 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.018038    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"]
Apr 24 23:59:42.120152 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.120115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3153a201-d81f-49d7-b7b9-7da26ac414fc-cert\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.120334 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.120160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5v2\" (UniqueName: \"kubernetes.io/projected/3153a201-d81f-49d7-b7b9-7da26ac414fc-kube-api-access-tt5v2\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.120334 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.120195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3153a201-d81f-49d7-b7b9-7da26ac414fc-manager-config\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.120334 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.120259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3153a201-d81f-49d7-b7b9-7da26ac414fc-metrics-cert\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.221318 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.221275    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5v2\" (UniqueName: \"kubernetes.io/projected/3153a201-d81f-49d7-b7b9-7da26ac414fc-kube-api-access-tt5v2\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.221497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.221330    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3153a201-d81f-49d7-b7b9-7da26ac414fc-manager-config\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.221497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.221400    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3153a201-d81f-49d7-b7b9-7da26ac414fc-metrics-cert\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.221497 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.221463    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3153a201-d81f-49d7-b7b9-7da26ac414fc-cert\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.222050 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.222020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3153a201-d81f-49d7-b7b9-7da26ac414fc-manager-config\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.224118 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.224100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3153a201-d81f-49d7-b7b9-7da26ac414fc-cert\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.224269 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.224247    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3153a201-d81f-49d7-b7b9-7da26ac414fc-metrics-cert\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.230418 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.230379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5v2\" (UniqueName: \"kubernetes.io/projected/3153a201-d81f-49d7-b7b9-7da26ac414fc-kube-api-access-tt5v2\") pod \"lws-controller-manager-58fbc56fdc-ql9rn\" (UID: \"3153a201-d81f-49d7-b7b9-7da26ac414fc\") " pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.315570 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.315482    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:42.445886 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:42.445862    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"]
Apr 24 23:59:43.102315 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:43.102276    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn" event={"ID":"3153a201-d81f-49d7-b7b9-7da26ac414fc","Type":"ContainerStarted","Data":"d725a7e104253c470b90646af2cb4a43809434ccc6095105d275a8975108aa28"}
Apr 24 23:59:45.110688 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:45.110650    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn" event={"ID":"3153a201-d81f-49d7-b7b9-7da26ac414fc","Type":"ContainerStarted","Data":"d644ba917cca6daed08b54734f09856dc9f0e14823449d92cf5e71b626dea4d2"}
Apr 24 23:59:45.111070 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:45.110794    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 24 23:59:45.128174 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:45.128126    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn" podStartSLOduration=1.6499838869999999 podStartE2EDuration="4.128110917s" podCreationTimestamp="2026-04-24 23:59:41 +0000 UTC" firstStartedPulling="2026-04-24 23:59:42.451382709 +0000 UTC m=+342.076422364" lastFinishedPulling="2026-04-24 23:59:44.929509737 +0000 UTC m=+344.554549394" observedRunningTime="2026-04-24 23:59:45.126089347 +0000 UTC m=+344.751129047" watchObservedRunningTime="2026-04-24 23:59:45.128110917 +0000 UTC m=+344.753150676"
Apr 24 23:59:56.118984 ip-10-0-134-82 kubenswrapper[2576]: I0424 23:59:56.118953    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58fbc56fdc-ql9rn"
Apr 25 00:00:05.509257 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.509220    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"]
Apr 25 00:00:05.512989 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.512971    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.515724 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.515703    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Apr 25 00:00:05.516021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.515989    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2ls7z\""
Apr 25 00:00:05.516143 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.516124    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Apr 25 00:00:05.523662 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.523643    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"]
Apr 25 00:00:05.630087 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.630050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f3d2b-2299-441e-b58b-dfb37bfe9713-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-f9rbg\" (UID: \"257f3d2b-2299-441e-b58b-dfb37bfe9713\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.630270 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.630168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqzk\" (UniqueName: \"kubernetes.io/projected/257f3d2b-2299-441e-b58b-dfb37bfe9713-kube-api-access-rzqzk\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-f9rbg\" (UID: \"257f3d2b-2299-441e-b58b-dfb37bfe9713\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.731699 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.731589    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f3d2b-2299-441e-b58b-dfb37bfe9713-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-f9rbg\" (UID: \"257f3d2b-2299-441e-b58b-dfb37bfe9713\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.731884 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.731790    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqzk\" (UniqueName: \"kubernetes.io/projected/257f3d2b-2299-441e-b58b-dfb37bfe9713-kube-api-access-rzqzk\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-f9rbg\" (UID: \"257f3d2b-2299-441e-b58b-dfb37bfe9713\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.732035 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.732010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/257f3d2b-2299-441e-b58b-dfb37bfe9713-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-f9rbg\" (UID: \"257f3d2b-2299-441e-b58b-dfb37bfe9713\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.747234 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.747208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqzk\" (UniqueName: \"kubernetes.io/projected/257f3d2b-2299-441e-b58b-dfb37bfe9713-kube-api-access-rzqzk\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-f9rbg\" (UID: \"257f3d2b-2299-441e-b58b-dfb37bfe9713\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.824116 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.824025    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:05.958434 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:05.958406    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"]
Apr 25 00:00:05.960870 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:00:05.960835    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257f3d2b_2299_441e_b58b_dfb37bfe9713.slice/crio-460c0b776cebea2889d2ab99f5b03be334132f758512c52027965edfcfc1d7ec WatchSource:0}: Error finding container 460c0b776cebea2889d2ab99f5b03be334132f758512c52027965edfcfc1d7ec: Status 404 returned error can't find the container with id 460c0b776cebea2889d2ab99f5b03be334132f758512c52027965edfcfc1d7ec
Apr 25 00:00:06.178969 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:06.178934    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg" event={"ID":"257f3d2b-2299-441e-b58b-dfb37bfe9713","Type":"ContainerStarted","Data":"460c0b776cebea2889d2ab99f5b03be334132f758512c52027965edfcfc1d7ec"}
Apr 25 00:00:16.218822 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:16.218782    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg" event={"ID":"257f3d2b-2299-441e-b58b-dfb37bfe9713","Type":"ContainerStarted","Data":"4d59137716597f1222f698c090703356dfd8bf6404ca6dfd044c21e5117c965f"}
Apr 25 00:00:16.219298 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:16.218912    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:00:16.240082 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:16.240029    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg" podStartSLOduration=1.630817212 podStartE2EDuration="11.240009697s" podCreationTimestamp="2026-04-25 00:00:05 +0000 UTC" firstStartedPulling="2026-04-25 00:00:05.963383556 +0000 UTC m=+365.588423211" lastFinishedPulling="2026-04-25 00:00:15.572576028 +0000 UTC m=+375.197615696" observedRunningTime="2026-04-25 00:00:16.237483411 +0000 UTC m=+375.862523105" watchObservedRunningTime="2026-04-25 00:00:16.240009697 +0000 UTC m=+375.865049374"
Apr 25 00:00:27.225462 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:00:27.225428    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-f9rbg"
Apr 25 00:01:01.942421 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:01.942383    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-m86g7"]
Apr 25 00:01:01.946878 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:01.946854    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:01.949283 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:01.949259    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6768d\""
Apr 25 00:01:01.949493 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:01.949480    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Apr 25 00:01:01.953453 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:01.953418    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-m86g7"]
Apr 25 00:01:01.987259 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:01.987227    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-m86g7"]
Apr 25 00:01:02.134900 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.134865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gxx\" (UniqueName: \"kubernetes.io/projected/e8f5d4e9-a22b-40b8-9683-5a7f58b1056f-kube-api-access-w8gxx\") pod \"limitador-limitador-67566c68b4-m86g7\" (UID: \"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f\") " pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.135097 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.134920    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8f5d4e9-a22b-40b8-9683-5a7f58b1056f-config-file\") pod \"limitador-limitador-67566c68b4-m86g7\" (UID: \"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f\") " pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.236468 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.236376    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gxx\" (UniqueName: \"kubernetes.io/projected/e8f5d4e9-a22b-40b8-9683-5a7f58b1056f-kube-api-access-w8gxx\") pod \"limitador-limitador-67566c68b4-m86g7\" (UID: \"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f\") " pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.236468 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.236424    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8f5d4e9-a22b-40b8-9683-5a7f58b1056f-config-file\") pod \"limitador-limitador-67566c68b4-m86g7\" (UID: \"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f\") " pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.237104 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.237083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8f5d4e9-a22b-40b8-9683-5a7f58b1056f-config-file\") pod \"limitador-limitador-67566c68b4-m86g7\" (UID: \"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f\") " pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.247425 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.247400    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gxx\" (UniqueName: \"kubernetes.io/projected/e8f5d4e9-a22b-40b8-9683-5a7f58b1056f-kube-api-access-w8gxx\") pod \"limitador-limitador-67566c68b4-m86g7\" (UID: \"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f\") " pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.259269 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.259242    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:02.393770 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:02.393735    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-m86g7"]
Apr 25 00:01:02.396906 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:01:02.396867    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f5d4e9_a22b_40b8_9683_5a7f58b1056f.slice/crio-4403aa591159dff6441a7e67bf5906041163608402fd1be2de0a95f856b492e9 WatchSource:0}: Error finding container 4403aa591159dff6441a7e67bf5906041163608402fd1be2de0a95f856b492e9: Status 404 returned error can't find the container with id 4403aa591159dff6441a7e67bf5906041163608402fd1be2de0a95f856b492e9
Apr 25 00:01:03.386471 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:03.386426    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7" event={"ID":"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f","Type":"ContainerStarted","Data":"4403aa591159dff6441a7e67bf5906041163608402fd1be2de0a95f856b492e9"}
Apr 25 00:01:07.404513 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:07.404469    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7" event={"ID":"e8f5d4e9-a22b-40b8-9683-5a7f58b1056f","Type":"ContainerStarted","Data":"649b8966fae965b8743df2a04f59193fed3ccdd2f548f0ca52c841d91796c153"}
Apr 25 00:01:07.404971 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:07.404639    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:01:07.420993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:07.420935    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7" podStartSLOduration=2.030627085 podStartE2EDuration="6.420917669s" podCreationTimestamp="2026-04-25 00:01:01 +0000 UTC" firstStartedPulling="2026-04-25 00:01:02.398930283 +0000 UTC m=+422.023969938" lastFinishedPulling="2026-04-25 00:01:06.789220869 +0000 UTC m=+426.414260522" observedRunningTime="2026-04-25 00:01:07.419711503 +0000 UTC m=+427.044751233" watchObservedRunningTime="2026-04-25 00:01:07.420917669 +0000 UTC m=+427.045957346"
Apr 25 00:01:18.409367 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:01:18.409338    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-m86g7"
Apr 25 00:03:54.654776 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.654738    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"]
Apr 25 00:03:54.658131 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.658113    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.661340 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.661314    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 25 00:03:54.661466 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.661429    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Apr 25 00:03:54.661535 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.661315    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4dwzn\""
Apr 25 00:03:54.661753 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.661735    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 25 00:03:54.669439 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.669400    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"]
Apr 25 00:03:54.784862 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.784831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.785051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.784877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.785051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.784899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.785051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.784955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.785051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.784979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.785051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.785012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bws7x\" (UniqueName: \"kubernetes.io/projected/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kube-api-access-bws7x\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.785051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.785051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885696    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885741    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885986 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885775    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885986 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885808    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885986 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885832    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bws7x\" (UniqueName: \"kubernetes.io/projected/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kube-api-access-bws7x\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885986 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885898    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.885986 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.885944    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.886231 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.886216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.886293 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.886241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.886363 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.886300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.886416 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.886384    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.888732 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.888705    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.888929 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.888913    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.893842 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.893822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bws7x\" (UniqueName: \"kubernetes.io/projected/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kube-api-access-bws7x\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:54.971535 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:54.971444    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:03:55.797581 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:55.797554    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"]
Apr 25 00:03:55.800471 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:03:55.800445    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8861e3b1_7544_4d4a_8aa3_bcd9a1f4fc09.slice/crio-e2226a300a1d5e981fd41f10e5fc0a4efbd8dabc7ecfc8a4f27f6cf88759b8ba WatchSource:0}: Error finding container e2226a300a1d5e981fd41f10e5fc0a4efbd8dabc7ecfc8a4f27f6cf88759b8ba: Status 404 returned error can't find the container with id e2226a300a1d5e981fd41f10e5fc0a4efbd8dabc7ecfc8a4f27f6cf88759b8ba
Apr 25 00:03:55.982801 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:03:55.982717    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" event={"ID":"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09","Type":"ContainerStarted","Data":"e2226a300a1d5e981fd41f10e5fc0a4efbd8dabc7ecfc8a4f27f6cf88759b8ba"}
Apr 25 00:04:00.004770 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:00.004734    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" event={"ID":"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09","Type":"ContainerStarted","Data":"30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa"}
Apr 25 00:04:00.798883 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:00.798851    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:04:00.799314 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:00.799293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:04:04.020665 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:04.020630    2576 generic.go:358] "Generic (PLEG): container finished" podID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerID="30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa" exitCode=0
Apr 25 00:04:04.021052 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:04.020689    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" event={"ID":"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09","Type":"ContainerDied","Data":"30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa"}
Apr 25 00:04:06.028795 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:06.028673    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" event={"ID":"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09","Type":"ContainerStarted","Data":"1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67"}
Apr 25 00:04:06.046822 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:06.046776    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" podStartSLOduration=2.801578561 podStartE2EDuration="12.046761003s" podCreationTimestamp="2026-04-25 00:03:54 +0000 UTC" firstStartedPulling="2026-04-25 00:03:55.802828373 +0000 UTC m=+595.427868026" lastFinishedPulling="2026-04-25 00:04:05.048010796 +0000 UTC m=+604.673050468" observedRunningTime="2026-04-25 00:04:06.045099354 +0000 UTC m=+605.670139040" watchObservedRunningTime="2026-04-25 00:04:06.046761003 +0000 UTC m=+605.671800678"
Apr 25 00:04:14.971726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:14.971630    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:04:14.972207 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:14.971784    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:04:14.984657 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:14.984623    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:04:15.070066 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:15.070035    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:04:28.452927 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.452890    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"]
Apr 25 00:04:28.453327 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.453224    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerName="main" containerID="cri-o://1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67" gracePeriod=30
Apr 25 00:04:28.711850 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.711782    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:04:28.800484 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800452    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tls-certs\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800677 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800504    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-home\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800677 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800539    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kserve-provision-location\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800677 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800569    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-model-cache\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800677 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800588    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-dshm\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800677 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800645    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bws7x\" (UniqueName: \"kubernetes.io/projected/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kube-api-access-bws7x\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800949 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800703    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tmp-dir\") pod \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\" (UID: \"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09\") "
Apr 25 00:04:28.800949 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800888    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-model-cache" (OuterVolumeSpecName: "model-cache") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:04:28.801043 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.800949    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-home" (OuterVolumeSpecName: "home") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:04:28.801043 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.801033    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:28.801150 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.801053    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:28.801150 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.801100    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:04:28.803015 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.802984    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:04:28.803015 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.803006    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-dshm" (OuterVolumeSpecName: "dshm") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:04:28.803160 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.803010    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kube-api-access-bws7x" (OuterVolumeSpecName: "kube-api-access-bws7x") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "kube-api-access-bws7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:04:28.862035 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.861990    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" (UID: "8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:04:28.901722 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.901686    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:28.901722 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.901715    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:28.901722 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.901728    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bws7x\" (UniqueName: \"kubernetes.io/projected/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-kube-api-access-bws7x\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:28.901945 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.901737    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:28.901945 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:28.901745    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:04:29.111803 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.111762    2576 generic.go:358] "Generic (PLEG): container finished" podID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerID="1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67" exitCode=0
Apr 25 00:04:29.112005 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.111832    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"
Apr 25 00:04:29.112005 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.111828    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" event={"ID":"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09","Type":"ContainerDied","Data":"1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67"}
Apr 25 00:04:29.112005 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.111941    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc" event={"ID":"8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09","Type":"ContainerDied","Data":"e2226a300a1d5e981fd41f10e5fc0a4efbd8dabc7ecfc8a4f27f6cf88759b8ba"}
Apr 25 00:04:29.112005 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.111963    2576 scope.go:117] "RemoveContainer" containerID="1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67"
Apr 25 00:04:29.120696 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.120671    2576 scope.go:117] "RemoveContainer" containerID="30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa"
Apr 25 00:04:29.153372 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.153336    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"]
Apr 25 00:04:29.164942 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.164907    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-b8d98c69-kf5rc"]
Apr 25 00:04:29.182040 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.182018    2576 scope.go:117] "RemoveContainer" containerID="1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67"
Apr 25 00:04:29.182412 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:29.182391    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67\": container with ID starting with 1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67 not found: ID does not exist" containerID="1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67"
Apr 25 00:04:29.182461 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.182421    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67"} err="failed to get container status \"1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67\": rpc error: code = NotFound desc = could not find container \"1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67\": container with ID starting with 1b81a90706698478332c73bff329833970c57e5e5ade3ed238b926a0d3e4df67 not found: ID does not exist"
Apr 25 00:04:29.182461 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.182441    2576 scope.go:117] "RemoveContainer" containerID="30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa"
Apr 25 00:04:29.182723 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:29.182702    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa\": container with ID starting with 30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa not found: ID does not exist" containerID="30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa"
Apr 25 00:04:29.182782 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:29.182730    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa"} err="failed to get container status \"30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa\": rpc error: code = NotFound desc = could not find container \"30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa\": container with ID starting with 30374cd9bb544fe37192ebbb39cb00d027f4e14a302bd7a1c8f0ec0924ad6eaa not found: ID does not exist"
Apr 25 00:04:30.880044 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:30.880010    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" path="/var/lib/kubelet/pods/8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09/volumes"
Apr 25 00:04:34.647282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.647249    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"]
Apr 25 00:04:34.647683 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.647654    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerName="main"
Apr 25 00:04:34.647683 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.647666    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerName="main"
Apr 25 00:04:34.647683 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.647677    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerName="storage-initializer"
Apr 25 00:04:34.647683 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.647682    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerName="storage-initializer"
Apr 25 00:04:34.647813 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.647738    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="8861e3b1-7544-4d4a-8aa3-bcd9a1f4fc09" containerName="main"
Apr 25 00:04:34.650926 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.650907    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.655950 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.655918    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 25 00:04:34.656083 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.656017    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 25 00:04:34.656161 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.656081    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4dwzn\""
Apr 25 00:04:34.656821 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.656800    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Apr 25 00:04:34.663181 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.663156    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"]
Apr 25 00:04:34.756691 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.756879 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdxb\" (UniqueName: \"kubernetes.io/projected/be226775-dd14-4307-afc6-e873072cd1d2-kube-api-access-7qdxb\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.756879 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.756879 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.756879 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.757010 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.757010 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.756922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.857775 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.857717    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.857803    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.857855    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdxb\" (UniqueName: \"kubernetes.io/projected/be226775-dd14-4307-afc6-e873072cd1d2-kube-api-access-7qdxb\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.857895    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.857920    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.857971    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.858005    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858254 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.858167    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858254 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.858198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858356 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.858328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.858389 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.858368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.860243 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.860225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.860535 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.860517    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.865796 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.865773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdxb\" (UniqueName: \"kubernetes.io/projected/be226775-dd14-4307-afc6-e873072cd1d2-kube-api-access-7qdxb\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:34.961457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:34.961371    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:04:35.091051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:35.091025    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"]
Apr 25 00:04:35.092967 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:04:35.092936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe226775_dd14_4307_afc6_e873072cd1d2.slice/crio-9f597c195aa330a7a018ca242ca8fd4ef22667fd8c4f98a7eabfed3c816791e4 WatchSource:0}: Error finding container 9f597c195aa330a7a018ca242ca8fd4ef22667fd8c4f98a7eabfed3c816791e4: Status 404 returned error can't find the container with id 9f597c195aa330a7a018ca242ca8fd4ef22667fd8c4f98a7eabfed3c816791e4
Apr 25 00:04:35.094736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:35.094719    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 25 00:04:35.135077 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:35.135045    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" event={"ID":"be226775-dd14-4307-afc6-e873072cd1d2","Type":"ContainerStarted","Data":"9f597c195aa330a7a018ca242ca8fd4ef22667fd8c4f98a7eabfed3c816791e4"}
Apr 25 00:04:36.147962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:36.147927    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" event={"ID":"be226775-dd14-4307-afc6-e873072cd1d2","Type":"ContainerStarted","Data":"f89b9bd89d3f30c38b5de01226c8e892d220e718582cf23c7f27354adc98fb05"}
Apr 25 00:04:37.248365 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:37.248329    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"]
Apr 25 00:04:37.280040 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:37.279996    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:37.280227 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:37.280084    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs podName:be226775-dd14-4307-afc6-e873072cd1d2 nodeName:}" failed. No retries permitted until 2026-04-25 00:04:37.780066458 +0000 UTC m=+637.405106113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" (UID: "be226775-dd14-4307-afc6-e873072cd1d2") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:37.785489 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:37.785451    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:37.785709 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:37.785528    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs podName:be226775-dd14-4307-afc6-e873072cd1d2 nodeName:}" failed. No retries permitted until 2026-04-25 00:04:38.785513195 +0000 UTC m=+638.410552850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" (UID: "be226775-dd14-4307-afc6-e873072cd1d2") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:38.154206 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:04:38.154168    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" podUID="be226775-dd14-4307-afc6-e873072cd1d2" containerName="storage-initializer" containerID="cri-o://f89b9bd89d3f30c38b5de01226c8e892d220e718582cf23c7f27354adc98fb05" gracePeriod=30
Apr 25 00:04:38.794932 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:38.794878    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:38.795396 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:38.794976    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs podName:be226775-dd14-4307-afc6-e873072cd1d2 nodeName:}" failed. No retries permitted until 2026-04-25 00:04:40.794956422 +0000 UTC m=+640.419996076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" (UID: "be226775-dd14-4307-afc6-e873072cd1d2") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:40.813351 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:40.813308    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:40.813855 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:40.813397    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs podName:be226775-dd14-4307-afc6-e873072cd1d2 nodeName:}" failed. No retries permitted until 2026-04-25 00:04:44.81337828 +0000 UTC m=+644.438417934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" (UID: "be226775-dd14-4307-afc6-e873072cd1d2") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:44.851511 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:44.851469    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:44.852008 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:44.851572    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs podName:be226775-dd14-4307-afc6-e873072cd1d2 nodeName:}" failed. No retries permitted until 2026-04-25 00:04:52.851552375 +0000 UTC m=+652.476592029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" (UID: "be226775-dd14-4307-afc6-e873072cd1d2") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:52.930722 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:52.930682    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:04:52.931186 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:04:52.930762    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs podName:be226775-dd14-4307-afc6-e873072cd1d2 nodeName:}" failed. No retries permitted until 2026-04-25 00:05:08.930745321 +0000 UTC m=+668.555784975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" (UID: "be226775-dd14-4307-afc6-e873072cd1d2") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Apr 25 00:05:00.339713 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.339675    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"]
Apr 25 00:05:00.342297 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.342278    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.344767 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.344745    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
Apr 25 00:05:00.353271 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.353243    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"]
Apr 25 00:05:00.400500 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.400702 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.400702 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.400702 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.400702 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.400965 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hvp\" (UniqueName: \"kubernetes.io/projected/605699ed-c332-4d7b-9731-dbe8cbf25f95-kube-api-access-82hvp\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.400965 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.400812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.501930 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.501893    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502128 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.501968    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82hvp\" (UniqueName: \"kubernetes.io/projected/605699ed-c332-4d7b-9731-dbe8cbf25f95-kube-api-access-82hvp\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502128 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.501993    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502128 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502029    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502128 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502105    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502343 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502144    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502343 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502174    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502491 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502413    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502491 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502465    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502563 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502498    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.502563 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.502535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.504462 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.504441    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.504898 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.504878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.509461 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.509440    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hvp\" (UniqueName: \"kubernetes.io/projected/605699ed-c332-4d7b-9731-dbe8cbf25f95-kube-api-access-82hvp\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.654416 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.654304    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:05:00.788208 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:00.788182    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"]
Apr 25 00:05:01.239212 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:01.239178    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" event={"ID":"605699ed-c332-4d7b-9731-dbe8cbf25f95","Type":"ContainerStarted","Data":"a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222"}
Apr 25 00:05:01.239212 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:01.239211    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" event={"ID":"605699ed-c332-4d7b-9731-dbe8cbf25f95","Type":"ContainerStarted","Data":"6a8241e42ac4b302f1b46d94d9ab67ba3a446d2cda8328b10fbcb873add85de4"}
Apr 25 00:05:08.272830 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.272799    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv_be226775-dd14-4307-afc6-e873072cd1d2/storage-initializer/0.log"
Apr 25 00:05:08.273296 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.272842    2576 generic.go:358] "Generic (PLEG): container finished" podID="be226775-dd14-4307-afc6-e873072cd1d2" containerID="f89b9bd89d3f30c38b5de01226c8e892d220e718582cf23c7f27354adc98fb05" exitCode=137
Apr 25 00:05:08.273296 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.272978    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" event={"ID":"be226775-dd14-4307-afc6-e873072cd1d2","Type":"ContainerDied","Data":"f89b9bd89d3f30c38b5de01226c8e892d220e718582cf23c7f27354adc98fb05"}
Apr 25 00:05:08.339313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.339284    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv_be226775-dd14-4307-afc6-e873072cd1d2/storage-initializer/0.log"
Apr 25 00:05:08.339453 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.339362    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:05:08.487073 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.486983    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-home\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487073 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487022    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-dshm\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487073 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487043    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487348 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487087    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-kserve-provision-location\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487348 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487102    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-tmp-dir\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487348 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487116    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-model-cache\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487348 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487165    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdxb\" (UniqueName: \"kubernetes.io/projected/be226775-dd14-4307-afc6-e873072cd1d2-kube-api-access-7qdxb\") pod \"be226775-dd14-4307-afc6-e873072cd1d2\" (UID: \"be226775-dd14-4307-afc6-e873072cd1d2\") "
Apr 25 00:05:08.487348 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487302    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-home" (OuterVolumeSpecName: "home") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:05:08.487569 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487429    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:05:08.487569 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487443    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:05:08.487569 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.487490    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:08.489442 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.489421    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be226775-dd14-4307-afc6-e873072cd1d2-kube-api-access-7qdxb" (OuterVolumeSpecName: "kube-api-access-7qdxb") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "kube-api-access-7qdxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:05:08.489841 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.489824    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-dshm" (OuterVolumeSpecName: "dshm") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:05:08.489901 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.489838    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:05:08.513221 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.513182    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be226775-dd14-4307-afc6-e873072cd1d2" (UID: "be226775-dd14-4307-afc6-e873072cd1d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:05:08.588977 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.588939    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdxb\" (UniqueName: \"kubernetes.io/projected/be226775-dd14-4307-afc6-e873072cd1d2-kube-api-access-7qdxb\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:08.588977 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.588971    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:08.588977 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.588985    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be226775-dd14-4307-afc6-e873072cd1d2-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:08.589212 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.588998    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:08.589212 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.589010    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:08.589212 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:08.589021    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be226775-dd14-4307-afc6-e873072cd1d2-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:05:09.277933 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:09.277859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv_be226775-dd14-4307-afc6-e873072cd1d2/storage-initializer/0.log"
Apr 25 00:05:09.278343 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:09.277932    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv" event={"ID":"be226775-dd14-4307-afc6-e873072cd1d2","Type":"ContainerDied","Data":"9f597c195aa330a7a018ca242ca8fd4ef22667fd8c4f98a7eabfed3c816791e4"}
Apr 25 00:05:09.278343 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:09.277964    2576 scope.go:117] "RemoveContainer" containerID="f89b9bd89d3f30c38b5de01226c8e892d220e718582cf23c7f27354adc98fb05"
Apr 25 00:05:09.278343 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:09.277984    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"
Apr 25 00:05:09.310635 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:09.310560    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"]
Apr 25 00:05:09.312495 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:09.312470    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5c5cf55f94plwxv"]
Apr 25 00:05:10.875240 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:05:10.875204    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be226775-dd14-4307-afc6-e873072cd1d2" path="/var/lib/kubelet/pods/be226775-dd14-4307-afc6-e873072cd1d2/volumes"
Apr 25 00:06:40.600786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:40.600749    2576 generic.go:358] "Generic (PLEG): container finished" podID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerID="a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222" exitCode=0
Apr 25 00:06:40.601196 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:40.600828    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" event={"ID":"605699ed-c332-4d7b-9731-dbe8cbf25f95","Type":"ContainerDied","Data":"a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222"}
Apr 25 00:06:41.606562 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:41.606527    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" event={"ID":"605699ed-c332-4d7b-9731-dbe8cbf25f95","Type":"ContainerStarted","Data":"0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114"}
Apr 25 00:06:41.624872 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:41.624827    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" podStartSLOduration=101.624811128 podStartE2EDuration="1m41.624811128s" podCreationTimestamp="2026-04-25 00:05:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:06:41.622812214 +0000 UTC m=+761.247851915" watchObservedRunningTime="2026-04-25 00:06:41.624811128 +0000 UTC m=+761.249850804"
Apr 25 00:06:50.655489 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:50.655453    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:06:50.655990 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:50.655504    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:06:50.667844 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:50.667814    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:06:51.653772 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:51.653736    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:06:52.585195 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:52.585163    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"]
Apr 25 00:06:52.833168 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:06:52.833130    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
Apr 25 00:06:52.833341 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:06:52.833219    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs podName:605699ed-c332-4d7b-9731-dbe8cbf25f95 nodeName:}" failed. No retries permitted until 2026-04-25 00:06:53.333201233 +0000 UTC m=+772.958240888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
Apr 25 00:06:53.337739 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:06:53.337705    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
Apr 25 00:06:53.337921 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:06:53.337788    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs podName:605699ed-c332-4d7b-9731-dbe8cbf25f95 nodeName:}" failed. No retries permitted until 2026-04-25 00:06:54.337772779 +0000 UTC m=+773.962812433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
Apr 25 00:06:53.649217 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:53.649128    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerName="main" containerID="cri-o://0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114" gracePeriod=30
Apr 25 00:06:53.900844 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:53.900788    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:06:54.042699 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.042661    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.042897 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.042734    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82hvp\" (UniqueName: \"kubernetes.io/projected/605699ed-c332-4d7b-9731-dbe8cbf25f95-kube-api-access-82hvp\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.042897 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.042757    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-tmp-dir\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.042897 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.042864    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-model-cache\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.043066 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.042930    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-home\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.043066 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.042962    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-dshm\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.043066 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043020    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-kserve-provision-location\") pod \"605699ed-c332-4d7b-9731-dbe8cbf25f95\" (UID: \"605699ed-c332-4d7b-9731-dbe8cbf25f95\") "
Apr 25 00:06:54.043066 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043024    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:06:54.043238 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043119    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-model-cache" (OuterVolumeSpecName: "model-cache") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:06:54.043238 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043203    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-home" (OuterVolumeSpecName: "home") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:06:54.043387 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043364    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.043387 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043388    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.043542 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.043401    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.045131 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.045108    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605699ed-c332-4d7b-9731-dbe8cbf25f95-kube-api-access-82hvp" (OuterVolumeSpecName: "kube-api-access-82hvp") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "kube-api-access-82hvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:06:54.045229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.045108    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:06:54.045412 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.045389    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-dshm" (OuterVolumeSpecName: "dshm") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:06:54.097271 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.097231    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "605699ed-c332-4d7b-9731-dbe8cbf25f95" (UID: "605699ed-c332-4d7b-9731-dbe8cbf25f95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:06:54.144567 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.144530    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.144567 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.144563    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/605699ed-c332-4d7b-9731-dbe8cbf25f95-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.144756 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.144575    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605699ed-c332-4d7b-9731-dbe8cbf25f95-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.144756 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.144584    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hvp\" (UniqueName: \"kubernetes.io/projected/605699ed-c332-4d7b-9731-dbe8cbf25f95-kube-api-access-82hvp\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:06:54.654026 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.653988    2576 generic.go:358] "Generic (PLEG): container finished" podID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerID="0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114" exitCode=0
Apr 25 00:06:54.654458 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.654050    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" event={"ID":"605699ed-c332-4d7b-9731-dbe8cbf25f95","Type":"ContainerDied","Data":"0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114"}
Apr 25 00:06:54.654458 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.654055    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"
Apr 25 00:06:54.654458 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.654079    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5" event={"ID":"605699ed-c332-4d7b-9731-dbe8cbf25f95","Type":"ContainerDied","Data":"6a8241e42ac4b302f1b46d94d9ab67ba3a446d2cda8328b10fbcb873add85de4"}
Apr 25 00:06:54.654458 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.654096    2576 scope.go:117] "RemoveContainer" containerID="0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114"
Apr 25 00:06:54.663156 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.663132    2576 scope.go:117] "RemoveContainer" containerID="a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222"
Apr 25 00:06:54.675746 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.675714    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"]
Apr 25 00:06:54.679435 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.679409    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-9f5964b54-2chh5"]
Apr 25 00:06:54.724915 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.724892    2576 scope.go:117] "RemoveContainer" containerID="0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114"
Apr 25 00:06:54.725269 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:06:54.725241    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114\": container with ID starting with 0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114 not found: ID does not exist" containerID="0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114"
Apr 25 00:06:54.725385 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.725276    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114"} err="failed to get container status \"0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114\": rpc error: code = NotFound desc = could not find container \"0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114\": container with ID starting with 0bfddc74cd04539911c7b7a62828047959bc54ca56713eae8edc6ad38d015114 not found: ID does not exist"
Apr 25 00:06:54.725385 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.725299    2576 scope.go:117] "RemoveContainer" containerID="a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222"
Apr 25 00:06:54.725616 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:06:54.725577    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222\": container with ID starting with a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222 not found: ID does not exist" containerID="a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222"
Apr 25 00:06:54.725696 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.725625    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222"} err="failed to get container status \"a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222\": rpc error: code = NotFound desc = could not find container \"a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222\": container with ID starting with a4f5c31c374d5e106b8aa2a8faa29147d8cb8c5e9129a3be479fc50900c0d222 not found: ID does not exist"
Apr 25 00:06:54.873908 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:54.873874    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" path="/var/lib/kubelet/pods/605699ed-c332-4d7b-9731-dbe8cbf25f95/volumes"
Apr 25 00:06:57.868923 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.868880    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"]
Apr 25 00:06:57.869577 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869553    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerName="main"
Apr 25 00:06:57.869726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869620    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerName="main"
Apr 25 00:06:57.869726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869638    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerName="storage-initializer"
Apr 25 00:06:57.869726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869647    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerName="storage-initializer"
Apr 25 00:06:57.869726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869678    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be226775-dd14-4307-afc6-e873072cd1d2" containerName="storage-initializer"
Apr 25 00:06:57.869726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869687    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be226775-dd14-4307-afc6-e873072cd1d2" containerName="storage-initializer"
Apr 25 00:06:57.870006 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869840    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="be226775-dd14-4307-afc6-e873072cd1d2" containerName="storage-initializer"
Apr 25 00:06:57.870006 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.869923    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="605699ed-c332-4d7b-9731-dbe8cbf25f95" containerName="main"
Apr 25 00:06:57.875363 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.875338    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"]
Apr 25 00:06:57.875509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.875478    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.877985 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.877958    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 25 00:06:57.878109 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.877981    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 25 00:06:57.878109 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.878076    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Apr 25 00:06:57.878390 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.878373    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4dwzn\""
Apr 25 00:06:57.977154 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.977154 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.977457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.977457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.977457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.977457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:57.977457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:57.977327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9n48\" (UniqueName: \"kubernetes.io/projected/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kube-api-access-b9n48\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077766 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077731    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077960 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077773    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077960 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077803    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077960 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077828    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077960 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077868    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077960 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077897    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.077960 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.077920    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b9n48\" (UniqueName: \"kubernetes.io/projected/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kube-api-access-b9n48\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.078249 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.078216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.078558 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.078508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.078558 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.078532    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.078883 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.078857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.080250 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.080229    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.080908 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.080882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.095587 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.095562    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9n48\" (UniqueName: \"kubernetes.io/projected/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kube-api-access-b9n48\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.186874 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.186772    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:06:58.318773 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.318744    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"]
Apr 25 00:06:58.321334 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:06:58.321306    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c56ed8a_6623_48d6_9978_3a9326bef2e6.slice/crio-536f73ef17ac3df7360628d7e0815c15c4619d4b23567cdfc6fb3f515b1027ae WatchSource:0}: Error finding container 536f73ef17ac3df7360628d7e0815c15c4619d4b23567cdfc6fb3f515b1027ae: Status 404 returned error can't find the container with id 536f73ef17ac3df7360628d7e0815c15c4619d4b23567cdfc6fb3f515b1027ae
Apr 25 00:06:58.671694 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.671661    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" event={"ID":"7c56ed8a-6623-48d6-9978-3a9326bef2e6","Type":"ContainerStarted","Data":"ed1c9674a675202ab170df219de576f81ae817fa48f6f4994c96555f8406eea5"}
Apr 25 00:06:58.671694 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:06:58.671695    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" event={"ID":"7c56ed8a-6623-48d6-9978-3a9326bef2e6","Type":"ContainerStarted","Data":"536f73ef17ac3df7360628d7e0815c15c4619d4b23567cdfc6fb3f515b1027ae"}
Apr 25 00:07:02.696182 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:02.696086    2576 generic.go:358] "Generic (PLEG): container finished" podID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerID="ed1c9674a675202ab170df219de576f81ae817fa48f6f4994c96555f8406eea5" exitCode=0
Apr 25 00:07:02.696182 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:02.696162    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" event={"ID":"7c56ed8a-6623-48d6-9978-3a9326bef2e6","Type":"ContainerDied","Data":"ed1c9674a675202ab170df219de576f81ae817fa48f6f4994c96555f8406eea5"}
Apr 25 00:07:30.818144 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:30.818047    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" event={"ID":"7c56ed8a-6623-48d6-9978-3a9326bef2e6","Type":"ContainerStarted","Data":"76bdf60bcd767248c96fcbc77f5cf9dca74de77f5a4f00a4f84281e012762d74"}
Apr 25 00:07:30.839898 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:30.839831    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podStartSLOduration=6.045522927 podStartE2EDuration="33.839813562s" podCreationTimestamp="2026-04-25 00:06:57 +0000 UTC" firstStartedPulling="2026-04-25 00:07:02.697278237 +0000 UTC m=+782.322317890" lastFinishedPulling="2026-04-25 00:07:30.49156887 +0000 UTC m=+810.116608525" observedRunningTime="2026-04-25 00:07:30.836461134 +0000 UTC m=+810.461500821" watchObservedRunningTime="2026-04-25 00:07:30.839813562 +0000 UTC m=+810.464853240"
Apr 25 00:07:38.187306 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:38.187258    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:07:38.187878 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:38.187339    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:07:38.188986 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:38.188931    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:07:48.193860 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:48.193813    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:07:58.156326 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.156289    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"]
Apr 25 00:07:58.162465 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.162443    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.164714 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.164692    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Apr 25 00:07:58.167914 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.167889    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"]
Apr 25 00:07:58.187572 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.187528    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:07:58.269173 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-dshm\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.269368 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tmp-dir\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.269368 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-model-cache\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.269368 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz57n\" (UniqueName: \"kubernetes.io/projected/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kube-api-access-mz57n\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.269368 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-home\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.269368 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.269570 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.269400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tls-certs\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.370777 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370727    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-model-cache\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.370962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370788    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mz57n\" (UniqueName: \"kubernetes.io/projected/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kube-api-access-mz57n\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.370962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370833    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-home\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.370962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370859    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.370962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370915    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tls-certs\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.370962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370951    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-dshm\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.371207 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.370992    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tmp-dir\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.371207 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.371135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-model-cache\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.371285 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.371201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-home\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.371285 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.371236    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.371384 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.371363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tmp-dir\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.373450 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.373428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-dshm\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.373670 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.373652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tls-certs\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.380356 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.380337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz57n\" (UniqueName: \"kubernetes.io/projected/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kube-api-access-mz57n\") pod \"precise-prefix-cache-test-kserve-74cfd44d4-77zv9\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.476330 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.476221    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:07:58.614413 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.614359    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"]
Apr 25 00:07:58.617105 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:07:58.617076    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731deaa1_ac55_482b_8d2a_78bd5bce2a8a.slice/crio-c0b9050873c06a2733573c533d948a724da5e0c26b13b531bd4d359551e35cc8 WatchSource:0}: Error finding container c0b9050873c06a2733573c533d948a724da5e0c26b13b531bd4d359551e35cc8: Status 404 returned error can't find the container with id c0b9050873c06a2733573c533d948a724da5e0c26b13b531bd4d359551e35cc8
Apr 25 00:07:58.923930 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.923871    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" event={"ID":"731deaa1-ac55-482b-8d2a-78bd5bce2a8a","Type":"ContainerStarted","Data":"d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b"}
Apr 25 00:07:58.923930 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:07:58.923918    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" event={"ID":"731deaa1-ac55-482b-8d2a-78bd5bce2a8a","Type":"ContainerStarted","Data":"c0b9050873c06a2733573c533d948a724da5e0c26b13b531bd4d359551e35cc8"}
Apr 25 00:08:03.948036 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:03.948002    2576 generic.go:358] "Generic (PLEG): container finished" podID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerID="d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b" exitCode=0
Apr 25 00:08:03.948436 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:03.948077    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" event={"ID":"731deaa1-ac55-482b-8d2a-78bd5bce2a8a","Type":"ContainerDied","Data":"d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b"}
Apr 25 00:08:04.952824 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:04.952790    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" event={"ID":"731deaa1-ac55-482b-8d2a-78bd5bce2a8a","Type":"ContainerStarted","Data":"086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5"}
Apr 25 00:08:04.971924 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:04.971853    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" podStartSLOduration=6.97183863 podStartE2EDuration="6.97183863s" podCreationTimestamp="2026-04-25 00:07:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:08:04.969963713 +0000 UTC m=+844.595003402" watchObservedRunningTime="2026-04-25 00:08:04.97183863 +0000 UTC m=+844.596878306"
Apr 25 00:08:08.187903 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:08.187855    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:08:08.477422 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:08.477333    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:08:08.477576 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:08.477445    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:08:08.490446 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:08.490413    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:08:08.979889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:08.979861    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:08:18.187531 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:18.187484    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:08:28.188308 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:28.188201    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:08:38.187980 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:38.187922    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:08:48.187850 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:48.187749    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:08:58.187931 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:08:58.187886    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Apr 25 00:09:00.284498 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.284464    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"]
Apr 25 00:09:00.284981 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.284848    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerName="main" containerID="cri-o://086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5" gracePeriod=30
Apr 25 00:09:00.539030 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.538962    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:09:00.582349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582317    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz57n\" (UniqueName: \"kubernetes.io/projected/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kube-api-access-mz57n\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582536 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582401    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-model-cache\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582536 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582444    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tls-certs\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582658 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582540    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-dshm\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582658 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582571    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kserve-provision-location\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582658 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582629    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tmp-dir\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582658 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582652    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-home\") pod \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\" (UID: \"731deaa1-ac55-482b-8d2a-78bd5bce2a8a\") "
Apr 25 00:09:00.582859 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582703    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:09:00.582995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582964    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.582995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582966    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:09:00.582995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.582973    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-home" (OuterVolumeSpecName: "home") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:09:00.585020 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.584989    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-dshm" (OuterVolumeSpecName: "dshm") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:09:00.585145 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.585120    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kube-api-access-mz57n" (OuterVolumeSpecName: "kube-api-access-mz57n") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "kube-api-access-mz57n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:09:00.585187 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.585163    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:09:00.637649 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.637561    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "731deaa1-ac55-482b-8d2a-78bd5bce2a8a" (UID: "731deaa1-ac55-482b-8d2a-78bd5bce2a8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:09:00.684142 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.684108    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz57n\" (UniqueName: \"kubernetes.io/projected/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kube-api-access-mz57n\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.684142 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.684136    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.684142 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.684148    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.684358 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.684157    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.684358 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.684165    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.684358 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.684173    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731deaa1-ac55-482b-8d2a-78bd5bce2a8a-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:09:00.835187 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.835155    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:09:00.836892 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:00.836871    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:09:01.163933 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.163835    2576 generic.go:358] "Generic (PLEG): container finished" podID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerID="086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5" exitCode=0
Apr 25 00:09:01.163933 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.163908    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" event={"ID":"731deaa1-ac55-482b-8d2a-78bd5bce2a8a","Type":"ContainerDied","Data":"086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5"}
Apr 25 00:09:01.164163 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.163936    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9" event={"ID":"731deaa1-ac55-482b-8d2a-78bd5bce2a8a","Type":"ContainerDied","Data":"c0b9050873c06a2733573c533d948a724da5e0c26b13b531bd4d359551e35cc8"}
Apr 25 00:09:01.164163 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.163944    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"
Apr 25 00:09:01.164163 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.163953    2576 scope.go:117] "RemoveContainer" containerID="086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5"
Apr 25 00:09:01.172471 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.172441    2576 scope.go:117] "RemoveContainer" containerID="d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b"
Apr 25 00:09:01.181931 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.181894    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"]
Apr 25 00:09:01.186189 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.186166    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-74cfd44d4-77zv9"]
Apr 25 00:09:01.242713 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.242673    2576 scope.go:117] "RemoveContainer" containerID="086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5"
Apr 25 00:09:01.243036 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:09:01.243016    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5\": container with ID starting with 086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5 not found: ID does not exist" containerID="086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5"
Apr 25 00:09:01.243090 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.243047    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5"} err="failed to get container status \"086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5\": rpc error: code = NotFound desc = could not find container \"086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5\": container with ID starting with 086b8092aeb9b8cae66d77f42789067b9077fde1643ed22176e34b7ee3eaece5 not found: ID does not exist"
Apr 25 00:09:01.243090 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.243069    2576 scope.go:117] "RemoveContainer" containerID="d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b"
Apr 25 00:09:01.243349 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:09:01.243326    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b\": container with ID starting with d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b not found: ID does not exist" containerID="d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b"
Apr 25 00:09:01.243394 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:01.243360    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b"} err="failed to get container status \"d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b\": rpc error: code = NotFound desc = could not find container \"d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b\": container with ID starting with d901e3e3b78579fe2d75d6e1264d0354627389555edc171a24b1e331de8e9e7b not found: ID does not exist"
Apr 25 00:09:02.875707 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:02.875663    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" path="/var/lib/kubelet/pods/731deaa1-ac55-482b-8d2a-78bd5bce2a8a/volumes"
Apr 25 00:09:04.357654 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.357618    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"]
Apr 25 00:09:04.358234 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.358210    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerName="storage-initializer"
Apr 25 00:09:04.358319 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.358239    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerName="storage-initializer"
Apr 25 00:09:04.358319 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.358262    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerName="main"
Apr 25 00:09:04.358319 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.358272    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerName="main"
Apr 25 00:09:04.358469 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.358371    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="731deaa1-ac55-482b-8d2a-78bd5bce2a8a" containerName="main"
Apr 25 00:09:04.362386 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.362360    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.364618 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.364578    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Apr 25 00:09:04.371353 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.371323    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"]
Apr 25 00:09:04.422067 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-model-cache\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.422067 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kserve-provision-location\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.422334 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tls-certs\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.422334 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-home\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.422334 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tmp-dir\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.422334 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-dshm\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.422483 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.422340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9vj\" (UniqueName: \"kubernetes.io/projected/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kube-api-access-4r9vj\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523618 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523559    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-model-cache\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523818 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523633    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kserve-provision-location\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523818 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523667    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tls-certs\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523818 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523699    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-home\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523818 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523721    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tmp-dir\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523818 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523762    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-dshm\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.523818 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.523798    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9vj\" (UniqueName: \"kubernetes.io/projected/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kube-api-access-4r9vj\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.524120 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.524039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-model-cache\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.524120 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.524066    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-home\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.524277 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.524260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tmp-dir\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.524342 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.524291    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kserve-provision-location\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.526310 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.526280    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-dshm\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.526580 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.526565    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tls-certs\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.532178 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.532150    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9vj\" (UniqueName: \"kubernetes.io/projected/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kube-api-access-4r9vj\") pod \"stop-feature-test-kserve-65cb4d7898-98nbk\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.675742 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.675590    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:04.813741 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:04.813710    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"]
Apr 25 00:09:04.816368 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:09:04.816336    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78efd0ec_7ff0_4b0a_9197_7f2a688cf0b0.slice/crio-61d5958b05c3879cd2169a01c6855ad547482fdbd1c81d3cf1861a54eba6d707 WatchSource:0}: Error finding container 61d5958b05c3879cd2169a01c6855ad547482fdbd1c81d3cf1861a54eba6d707: Status 404 returned error can't find the container with id 61d5958b05c3879cd2169a01c6855ad547482fdbd1c81d3cf1861a54eba6d707
Apr 25 00:09:05.182238 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:05.182197    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" event={"ID":"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0","Type":"ContainerStarted","Data":"26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a"}
Apr 25 00:09:05.182238 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:05.182240    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" event={"ID":"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0","Type":"ContainerStarted","Data":"61d5958b05c3879cd2169a01c6855ad547482fdbd1c81d3cf1861a54eba6d707"}
Apr 25 00:09:08.197765 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:08.197732    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:09:08.206064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:08.206037    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:09:13.218774 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:13.218736    2576 generic.go:358] "Generic (PLEG): container finished" podID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerID="26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a" exitCode=0
Apr 25 00:09:13.219218 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:13.218794    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" event={"ID":"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0","Type":"ContainerDied","Data":"26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a"}
Apr 25 00:09:14.227159 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:14.227124    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" event={"ID":"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0","Type":"ContainerStarted","Data":"4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac"}
Apr 25 00:09:14.248084 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:14.248022    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podStartSLOduration=10.248002624 podStartE2EDuration="10.248002624s" podCreationTimestamp="2026-04-25 00:09:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:09:14.244973107 +0000 UTC m=+913.870012803" watchObservedRunningTime="2026-04-25 00:09:14.248002624 +0000 UTC m=+913.873042302"
Apr 25 00:09:14.676113 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:14.676071    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:14.676294 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:14.676129    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:09:14.677582 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:14.677549    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:09:24.676410 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:24.676352    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:09:34.676792 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:34.676741    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:09:36.318210 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:36.318172    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"]
Apr 25 00:09:36.318656 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:36.318560    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main" containerID="cri-o://76bdf60bcd767248c96fcbc77f5cf9dca74de77f5a4f00a4f84281e012762d74" gracePeriod=30
Apr 25 00:09:40.866082 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.866047    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"]
Apr 25 00:09:40.873545 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.873513    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.875879 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.875850    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Apr 25 00:09:40.881414 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.881387    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"]
Apr 25 00:09:40.976135 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-model-cache\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.976135 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-home\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.976400 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-tmp-dir\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.976400 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsr8t\" (UniqueName: \"kubernetes.io/projected/6d2048fb-612c-4874-bc5a-26b9f87f60da-kube-api-access-nsr8t\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.976400 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2048fb-612c-4874-bc5a-26b9f87f60da-tls-certs\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.976560 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-dshm\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:40.976560 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:40.976538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077311 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077273    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077311 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077322    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-model-cache\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077347    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-home\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077488    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-tmp-dir\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077723 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077647    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nsr8t\" (UniqueName: \"kubernetes.io/projected/6d2048fb-612c-4874-bc5a-26b9f87f60da-kube-api-access-nsr8t\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077723 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077704    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2048fb-612c-4874-bc5a-26b9f87f60da-tls-certs\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077825 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077731    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-dshm\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077825 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-model-cache\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077825 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.077972 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.077943    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-tmp-dir\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.078102 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.078082    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-home\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.080034 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.080013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-dshm\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.080480 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.080457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2048fb-612c-4874-bc5a-26b9f87f60da-tls-certs\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.088164 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.088140    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsr8t\" (UniqueName: \"kubernetes.io/projected/6d2048fb-612c-4874-bc5a-26b9f87f60da-kube-api-access-nsr8t\") pod \"custom-route-timeout-test-kserve-6bf947bfc8-lrh5g\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.185877 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.185784    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:09:41.323398 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.323363    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"]
Apr 25 00:09:41.326552 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:09:41.326522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2048fb_612c_4874_bc5a_26b9f87f60da.slice/crio-c4443fb29cbb2101bb1ad016e7652681a91a612ef9668ad98675686cdf4cca9b WatchSource:0}: Error finding container c4443fb29cbb2101bb1ad016e7652681a91a612ef9668ad98675686cdf4cca9b: Status 404 returned error can't find the container with id c4443fb29cbb2101bb1ad016e7652681a91a612ef9668ad98675686cdf4cca9b
Apr 25 00:09:41.328636 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:41.328576    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 25 00:09:42.334021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:42.333979    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" event={"ID":"6d2048fb-612c-4874-bc5a-26b9f87f60da","Type":"ContainerStarted","Data":"a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40"}
Apr 25 00:09:42.334466 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:42.334032    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" event={"ID":"6d2048fb-612c-4874-bc5a-26b9f87f60da","Type":"ContainerStarted","Data":"c4443fb29cbb2101bb1ad016e7652681a91a612ef9668ad98675686cdf4cca9b"}
Apr 25 00:09:44.676388 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:44.676332    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:09:54.676724 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:09:54.676679    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:10:04.676230 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:04.676171    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:10:06.425889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.425859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf_7c56ed8a-6623-48d6-9978-3a9326bef2e6/main/0.log"
Apr 25 00:10:06.426305 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.426279    2576 generic.go:358] "Generic (PLEG): container finished" podID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerID="76bdf60bcd767248c96fcbc77f5cf9dca74de77f5a4f00a4f84281e012762d74" exitCode=137
Apr 25 00:10:06.426375 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.426341    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" event={"ID":"7c56ed8a-6623-48d6-9978-3a9326bef2e6","Type":"ContainerDied","Data":"76bdf60bcd767248c96fcbc77f5cf9dca74de77f5a4f00a4f84281e012762d74"}
Apr 25 00:10:06.557864 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.557833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf_7c56ed8a-6623-48d6-9978-3a9326bef2e6/main/0.log"
Apr 25 00:10:06.558222 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.558206    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:10:06.623313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623278    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9n48\" (UniqueName: \"kubernetes.io/projected/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kube-api-access-b9n48\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623495 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623339    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kserve-provision-location\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623495 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623365    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-model-cache\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623590 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623472    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-dshm\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623590 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623535    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-home\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623590 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623576    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tmp-dir\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623758 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623657    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:10:06.623758 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623686    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tls-certs\") pod \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\" (UID: \"7c56ed8a-6623-48d6-9978-3a9326bef2e6\") "
Apr 25 00:10:06.623972 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.623956    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:06.624269 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.624224    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-home" (OuterVolumeSpecName: "home") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:10:06.626123 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.626087    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-dshm" (OuterVolumeSpecName: "dshm") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:10:06.626233 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.626190    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kube-api-access-b9n48" (OuterVolumeSpecName: "kube-api-access-b9n48") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "kube-api-access-b9n48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:10:06.626292 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.626258    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:10:06.637099 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.637066    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:10:06.679868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.679820    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c56ed8a-6623-48d6-9978-3a9326bef2e6" (UID: "7c56ed8a-6623-48d6-9978-3a9326bef2e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:10:06.724889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.724855    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:06.725051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.724922    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:06.725051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.724939    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c56ed8a-6623-48d6-9978-3a9326bef2e6-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:06.725051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.724956    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9n48\" (UniqueName: \"kubernetes.io/projected/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kube-api-access-b9n48\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:06.725051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.724973    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:06.725051 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:06.724989    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c56ed8a-6623-48d6-9978-3a9326bef2e6-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:10:07.434947 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.434893    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf_7c56ed8a-6623-48d6-9978-3a9326bef2e6/main/0.log"
Apr 25 00:10:07.435582 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.435544    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf" event={"ID":"7c56ed8a-6623-48d6-9978-3a9326bef2e6","Type":"ContainerDied","Data":"536f73ef17ac3df7360628d7e0815c15c4619d4b23567cdfc6fb3f515b1027ae"}
Apr 25 00:10:07.435730 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.435648    2576 scope.go:117] "RemoveContainer" containerID="76bdf60bcd767248c96fcbc77f5cf9dca74de77f5a4f00a4f84281e012762d74"
Apr 25 00:10:07.435730 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.435717    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"
Apr 25 00:10:07.448286 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.448263    2576 scope.go:117] "RemoveContainer" containerID="ed1c9674a675202ab170df219de576f81ae817fa48f6f4994c96555f8406eea5"
Apr 25 00:10:07.455468 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.455438    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"]
Apr 25 00:10:07.460764 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:07.460740    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-d65657bc4-bxcwf"]
Apr 25 00:10:08.874851 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:08.874809    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" path="/var/lib/kubelet/pods/7c56ed8a-6623-48d6-9978-3a9326bef2e6/volumes"
Apr 25 00:10:14.677058 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:14.676965    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:10:24.676659 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:24.676592    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:10:34.676479 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:34.676431    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:10:44.676319 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:44.676271    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Apr 25 00:10:52.608387 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:52.608351    2576 generic.go:358] "Generic (PLEG): container finished" podID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerID="a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40" exitCode=0
Apr 25 00:10:52.608825 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:52.608414    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" event={"ID":"6d2048fb-612c-4874-bc5a-26b9f87f60da","Type":"ContainerDied","Data":"a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40"}
Apr 25 00:10:53.614268 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:53.614220    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" event={"ID":"6d2048fb-612c-4874-bc5a-26b9f87f60da","Type":"ContainerStarted","Data":"4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504"}
Apr 25 00:10:53.634032 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:53.633977    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podStartSLOduration=73.633959628 podStartE2EDuration="1m13.633959628s" podCreationTimestamp="2026-04-25 00:09:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:10:53.632456403 +0000 UTC m=+1013.257496081" watchObservedRunningTime="2026-04-25 00:10:53.633959628 +0000 UTC m=+1013.258999372"
Apr 25 00:10:54.686748 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:54.686714    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:10:54.694567 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:54.694532    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:10:55.742375 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:55.742341    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"]
Apr 25 00:10:56.632006 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:10:56.631960    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main" containerID="cri-o://4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac" gracePeriod=30
Apr 25 00:11:01.186234 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:01.186150    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:11:01.186234 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:01.186206    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:11:01.187774 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:01.187742    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:11:11.186938 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:11.186889    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:11:18.745947 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.745881    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"]
Apr 25 00:11:18.746510 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.746488    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="storage-initializer"
Apr 25 00:11:18.746614 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.746512    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="storage-initializer"
Apr 25 00:11:18.746614 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.746537    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main"
Apr 25 00:11:18.746614 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.746545    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main"
Apr 25 00:11:18.746771 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.746652    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c56ed8a-6623-48d6-9978-3a9326bef2e6" containerName="main"
Apr 25 00:11:18.749851 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.749823    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.760873 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.760843    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"]
Apr 25 00:11:18.906974 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.906929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-home\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.906974 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.906978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13974f39-01b9-4b0e-928e-7aefc9ff940f-tls-certs\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.907201 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.907002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-model-cache\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.907201 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.907039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-dshm\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.907201 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.907105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-kserve-provision-location\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.907201 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.907127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jwn\" (UniqueName: \"kubernetes.io/projected/13974f39-01b9-4b0e-928e-7aefc9ff940f-kube-api-access-k4jwn\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:18.907350 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:18.907221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-tmp-dir\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008451 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008355    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-home\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008451 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008400    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13974f39-01b9-4b0e-928e-7aefc9ff940f-tls-certs\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008719 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008579    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-model-cache\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008797 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008776    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-dshm\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008859 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008839    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-kserve-provision-location\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008910 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-home\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.008910 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008874    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jwn\" (UniqueName: \"kubernetes.io/projected/13974f39-01b9-4b0e-928e-7aefc9ff940f-kube-api-access-k4jwn\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.009016 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008944    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-tmp-dir\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.009016 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.008977    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-model-cache\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.009244 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.009206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-kserve-provision-location\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.009358 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.009286    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-tmp-dir\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.011174 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.011152    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13974f39-01b9-4b0e-928e-7aefc9ff940f-dshm\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.011579 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.011563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13974f39-01b9-4b0e-928e-7aefc9ff940f-tls-certs\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.016251 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.016228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jwn\" (UniqueName: \"kubernetes.io/projected/13974f39-01b9-4b0e-928e-7aefc9ff940f-kube-api-access-k4jwn\") pod \"stop-feature-test-kserve-65cb4d7898-bdr5z\" (UID: \"13974f39-01b9-4b0e-928e-7aefc9ff940f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.062470 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.062426    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"
Apr 25 00:11:19.201082 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.201052    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z"]
Apr 25 00:11:19.203351 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:11:19.203314    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13974f39_01b9_4b0e_928e_7aefc9ff940f.slice/crio-7b72a911660d6139b1f8c79dcd11d795c3e11dc000287509fc78397a0f2886da WatchSource:0}: Error finding container 7b72a911660d6139b1f8c79dcd11d795c3e11dc000287509fc78397a0f2886da: Status 404 returned error can't find the container with id 7b72a911660d6139b1f8c79dcd11d795c3e11dc000287509fc78397a0f2886da
Apr 25 00:11:19.716675 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.716632    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z" event={"ID":"13974f39-01b9-4b0e-928e-7aefc9ff940f","Type":"ContainerStarted","Data":"f63b2a9f8e8859761f6fcc7f2f052e5bd0d274c699b45f95c716c6839f106bc0"}
Apr 25 00:11:19.716675 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:19.716686    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-bdr5z" event={"ID":"13974f39-01b9-4b0e-928e-7aefc9ff940f","Type":"ContainerStarted","Data":"7b72a911660d6139b1f8c79dcd11d795c3e11dc000287509fc78397a0f2886da"}
Apr 25 00:11:21.186993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:21.186951    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:11:26.900149 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.900120    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-98nbk_78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0/main/0.log"
Apr 25 00:11:26.900649 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.900496    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:11:26.981463 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981422    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r9vj\" (UniqueName: \"kubernetes.io/projected/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kube-api-access-4r9vj\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.981463 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981465    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tls-certs\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.981736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981486    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-home\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.981736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981528    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tmp-dir\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.981736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981562    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-dshm\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.981736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981638    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-model-cache\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.981736 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.981667    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kserve-provision-location\") pod \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\" (UID: \"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0\") "
Apr 25 00:11:26.982239 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.982133    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:11:26.982380 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.982258    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-home" (OuterVolumeSpecName: "home") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:11:26.984304 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.984280    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:11:26.984431 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.984357    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kube-api-access-4r9vj" (OuterVolumeSpecName: "kube-api-access-4r9vj") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "kube-api-access-4r9vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:11:26.984431 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.984399    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-dshm" (OuterVolumeSpecName: "dshm") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:11:26.994103 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:26.994065    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:11:27.038840 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.038793    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" (UID: "78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:11:27.082433 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082385    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9vj\" (UniqueName: \"kubernetes.io/projected/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kube-api-access-4r9vj\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.082433 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082427    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.082433 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082442    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.082695 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082454    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.082695 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082466    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.082695 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082476    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.082695 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.082487    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:11:27.747856 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.747821    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-98nbk_78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0/main/0.log"
Apr 25 00:11:27.748212 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.748184    2576 generic.go:358] "Generic (PLEG): container finished" podID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerID="4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac" exitCode=137
Apr 25 00:11:27.748284 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.748259    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"
Apr 25 00:11:27.748284 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.748266    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" event={"ID":"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0","Type":"ContainerDied","Data":"4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac"}
Apr 25 00:11:27.748385 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.748309    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk" event={"ID":"78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0","Type":"ContainerDied","Data":"61d5958b05c3879cd2169a01c6855ad547482fdbd1c81d3cf1861a54eba6d707"}
Apr 25 00:11:27.748385 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.748333    2576 scope.go:117] "RemoveContainer" containerID="4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac"
Apr 25 00:11:27.757644 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.757626    2576 scope.go:117] "RemoveContainer" containerID="26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a"
Apr 25 00:11:27.771699 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.771667    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"]
Apr 25 00:11:27.774851 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.774829    2576 scope.go:117] "RemoveContainer" containerID="4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac"
Apr 25 00:11:27.775205 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:11:27.775183    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac\": container with ID starting with 4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac not found: ID does not exist" containerID="4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac"
Apr 25 00:11:27.775308 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.775216    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac"} err="failed to get container status \"4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac\": rpc error: code = NotFound desc = could not find container \"4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac\": container with ID starting with 4af44f06d7d28bcbd29c96a176eede062b642ae7052b3c1577bf319abac99dac not found: ID does not exist"
Apr 25 00:11:27.775308 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.775241    2576 scope.go:117] "RemoveContainer" containerID="26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a"
Apr 25 00:11:27.775591 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.775536    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-65cb4d7898-98nbk"]
Apr 25 00:11:27.775591 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:11:27.775549    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a\": container with ID starting with 26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a not found: ID does not exist" containerID="26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a"
Apr 25 00:11:27.775776 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:27.775581    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a"} err="failed to get container status \"26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a\": rpc error: code = NotFound desc = could not find container \"26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a\": container with ID starting with 26ac7cffc22ae95804443614ea4348c2454043af7d92c74bf0fa146661944c1a not found: ID does not exist"
Apr 25 00:11:28.873697 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:28.873653    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" path="/var/lib/kubelet/pods/78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0/volumes"
Apr 25 00:11:31.187207 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:31.187163    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:11:41.186523 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:41.186478    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:11:51.186527 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:11:51.186436    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:12:01.186762 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:01.186656    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:12:11.187089 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:11.187038    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:12:21.186390 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:21.186335    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Apr 25 00:12:31.196417 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:31.196379    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:12:31.204243 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:31.204217    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:12:37.177792 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:37.177751    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"]
Apr 25 00:12:37.178392 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:37.178125    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main" containerID="cri-o://4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504" gracePeriod=30
Apr 25 00:12:55.851313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.851280    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"]
Apr 25 00:12:55.851786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.851676    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="storage-initializer"
Apr 25 00:12:55.851786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.851688    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="storage-initializer"
Apr 25 00:12:55.851786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.851695    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main"
Apr 25 00:12:55.851786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.851701    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main"
Apr 25 00:12:55.851918 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.851801    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="78efd0ec-7ff0-4b0a-9197-7f2a688cf0b0" containerName="main"
Apr 25 00:12:55.855188 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.855167    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.857614 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.857575    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Apr 25 00:12:55.864971 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.864943    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"]
Apr 25 00:12:55.905050 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-kserve-provision-location\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.905254 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c310a32-fd3c-406e-a302-2525e3803c40-tls-certs\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.905254 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56bx\" (UniqueName: \"kubernetes.io/projected/2c310a32-fd3c-406e-a302-2525e3803c40-kube-api-access-c56bx\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.905254 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-home\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.905432 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-dshm\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.905432 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-model-cache\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:55.905432 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:55.905353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-tmp-dir\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006193 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006152    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c310a32-fd3c-406e-a302-2525e3803c40-tls-certs\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006408 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006208    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c56bx\" (UniqueName: \"kubernetes.io/projected/2c310a32-fd3c-406e-a302-2525e3803c40-kube-api-access-c56bx\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006408 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006263    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-home\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006408 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006280    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-dshm\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006408 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006313    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-model-cache\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006408 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006337    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-tmp-dir\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.006726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006462    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-kserve-provision-location\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.007000 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.006970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-home\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.007099 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.007035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-model-cache\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.007099 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.007051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-tmp-dir\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.007205 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.007189    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-kserve-provision-location\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.009094 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.009061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-dshm\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.009299 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.009278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c310a32-fd3c-406e-a302-2525e3803c40-tls-certs\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.014038 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.014005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56bx\" (UniqueName: \"kubernetes.io/projected/2c310a32-fd3c-406e-a302-2525e3803c40-kube-api-access-c56bx\") pod \"router-with-refs-test-kserve-7d795bc9b6-7tbfw\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.167323 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.167235    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:12:56.298815 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:56.298788    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"]
Apr 25 00:12:56.301300 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:12:56.301268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c310a32_fd3c_406e_a302_2525e3803c40.slice/crio-d52af229a3ab22733ad6495d38b6d0de44906a7c03ab23dbf9cb63c7e3db7a7a WatchSource:0}: Error finding container d52af229a3ab22733ad6495d38b6d0de44906a7c03ab23dbf9cb63c7e3db7a7a: Status 404 returned error can't find the container with id d52af229a3ab22733ad6495d38b6d0de44906a7c03ab23dbf9cb63c7e3db7a7a
Apr 25 00:12:57.069619 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:57.069558    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" event={"ID":"2c310a32-fd3c-406e-a302-2525e3803c40","Type":"ContainerStarted","Data":"cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e"}
Apr 25 00:12:57.069619 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:12:57.069593    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" event={"ID":"2c310a32-fd3c-406e-a302-2525e3803c40","Type":"ContainerStarted","Data":"d52af229a3ab22733ad6495d38b6d0de44906a7c03ab23dbf9cb63c7e3db7a7a"}
Apr 25 00:13:07.448458 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.448430    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6bf947bfc8-lrh5g_6d2048fb-612c-4874-bc5a-26b9f87f60da/main/0.log"
Apr 25 00:13:07.448810 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.448789    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:13:07.515334 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515297    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsr8t\" (UniqueName: \"kubernetes.io/projected/6d2048fb-612c-4874-bc5a-26b9f87f60da-kube-api-access-nsr8t\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515505 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515359    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-tmp-dir\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515505 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515377    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-kserve-provision-location\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515654 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515502    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2048fb-612c-4874-bc5a-26b9f87f60da-tls-certs\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515654 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515565    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-model-cache\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515654 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515634    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-home\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515807 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515662    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-dshm\") pod \"6d2048fb-612c-4874-bc5a-26b9f87f60da\" (UID: \"6d2048fb-612c-4874-bc5a-26b9f87f60da\") "
Apr 25 00:13:07.515887 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.515860    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:13:07.516043 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.516026    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:07.516314 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.516290    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-home" (OuterVolumeSpecName: "home") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:13:07.518113 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.518090    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-dshm" (OuterVolumeSpecName: "dshm") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:13:07.518220 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.518185    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2048fb-612c-4874-bc5a-26b9f87f60da-kube-api-access-nsr8t" (OuterVolumeSpecName: "kube-api-access-nsr8t") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "kube-api-access-nsr8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:13:07.518263 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.518218    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2048fb-612c-4874-bc5a-26b9f87f60da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:13:07.528762 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.528740    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:13:07.575965 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.575923    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d2048fb-612c-4874-bc5a-26b9f87f60da" (UID: "6d2048fb-612c-4874-bc5a-26b9f87f60da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:13:07.616911 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.616877    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:07.616911 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.616909    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:07.617104 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.616920    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2048fb-612c-4874-bc5a-26b9f87f60da-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:07.617104 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.616930    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:07.617104 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.616938    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2048fb-612c-4874-bc5a-26b9f87f60da-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:07.617104 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:07.616946    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsr8t\" (UniqueName: \"kubernetes.io/projected/6d2048fb-612c-4874-bc5a-26b9f87f60da-kube-api-access-nsr8t\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:13:08.106571 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.106539    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6bf947bfc8-lrh5g_6d2048fb-612c-4874-bc5a-26b9f87f60da/main/0.log"
Apr 25 00:13:08.106927 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.106900    2576 generic.go:358] "Generic (PLEG): container finished" podID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerID="4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504" exitCode=137
Apr 25 00:13:08.107076 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.106981    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"
Apr 25 00:13:08.107076 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.106990    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" event={"ID":"6d2048fb-612c-4874-bc5a-26b9f87f60da","Type":"ContainerDied","Data":"4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504"}
Apr 25 00:13:08.107076 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.107039    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g" event={"ID":"6d2048fb-612c-4874-bc5a-26b9f87f60da","Type":"ContainerDied","Data":"c4443fb29cbb2101bb1ad016e7652681a91a612ef9668ad98675686cdf4cca9b"}
Apr 25 00:13:08.107076 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.107060    2576 scope.go:117] "RemoveContainer" containerID="4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504"
Apr 25 00:13:08.115974 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.115952    2576 scope.go:117] "RemoveContainer" containerID="a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40"
Apr 25 00:13:08.129378 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.129349    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"]
Apr 25 00:13:08.131353 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.131335    2576 scope.go:117] "RemoveContainer" containerID="4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504"
Apr 25 00:13:08.131700 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:13:08.131679    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504\": container with ID starting with 4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504 not found: ID does not exist" containerID="4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504"
Apr 25 00:13:08.131778 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.131709    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504"} err="failed to get container status \"4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504\": rpc error: code = NotFound desc = could not find container \"4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504\": container with ID starting with 4e2c8645ae6eb35f45a0ab9f14453dd6f29c08a4f7c63f6709ec42c728bf0504 not found: ID does not exist"
Apr 25 00:13:08.131778 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.131728    2576 scope.go:117] "RemoveContainer" containerID="a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40"
Apr 25 00:13:08.132013 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:13:08.131986    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40\": container with ID starting with a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40 not found: ID does not exist" containerID="a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40"
Apr 25 00:13:08.132107 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.132021    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40"} err="failed to get container status \"a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40\": rpc error: code = NotFound desc = could not find container \"a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40\": container with ID starting with a57f4a20bf38cfbd01a438f103b629f8a60176ecf29d7ff8028a4db939e9bd40 not found: ID does not exist"
Apr 25 00:13:08.133205 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.133187    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bf947bfc8-lrh5g"]
Apr 25 00:13:08.874234 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:13:08.874199    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" path="/var/lib/kubelet/pods/6d2048fb-612c-4874-bc5a-26b9f87f60da/volumes"
Apr 25 00:14:00.860377 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:00.860348    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:14:00.862271 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:00.862246    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:14:13.333969 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:13.333931    2576 generic.go:358] "Generic (PLEG): container finished" podID="2c310a32-fd3c-406e-a302-2525e3803c40" containerID="cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e" exitCode=0
Apr 25 00:14:13.334430 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:13.334003    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" event={"ID":"2c310a32-fd3c-406e-a302-2525e3803c40","Type":"ContainerDied","Data":"cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e"}
Apr 25 00:14:14.339279 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:14.339237    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" event={"ID":"2c310a32-fd3c-406e-a302-2525e3803c40","Type":"ContainerStarted","Data":"163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2"}
Apr 25 00:14:14.358021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:14.357967    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podStartSLOduration=79.357953924 podStartE2EDuration="1m19.357953924s" podCreationTimestamp="2026-04-25 00:12:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:14:14.356889991 +0000 UTC m=+1213.981929686" watchObservedRunningTime="2026-04-25 00:14:14.357953924 +0000 UTC m=+1213.982993606"
Apr 25 00:14:16.167452 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:16.167409    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:14:16.167850 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:16.167467    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:14:16.168759 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:16.168729    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:14:26.168463 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:26.168408    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:14:36.167786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:36.167743    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:14:46.168374 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:46.168284    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:14:56.168428 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:14:56.168380    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:15:06.167933 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:06.167884    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:15:16.168231 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:16.168189    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:15:26.168519 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:26.168468    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:15:36.167719 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:36.167661    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:15:46.167789 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:46.167743    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Apr 25 00:15:56.178248 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:56.178217    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:15:56.186385 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:15:56.186342    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:16:02.473283 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:02.473244    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"]
Apr 25 00:16:02.473703 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:02.473569    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main" containerID="cri-o://163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2" gracePeriod=30
Apr 25 00:16:24.304550 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.304513    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"]
Apr 25 00:16:24.305021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.304954    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="storage-initializer"
Apr 25 00:16:24.305021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.304968    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="storage-initializer"
Apr 25 00:16:24.305021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.304986    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main"
Apr 25 00:16:24.305021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.304992    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main"
Apr 25 00:16:24.305146 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.305048    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="6d2048fb-612c-4874-bc5a-26b9f87f60da" containerName="main"
Apr 25 00:16:24.308044 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.308024    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.310426 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.310405    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Apr 25 00:16:24.310626 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.310522    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-75j2j\""
Apr 25 00:16:24.320715 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.320690    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"]
Apr 25 00:16:24.322399 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.322373    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"]
Apr 25 00:16:24.326310 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.326291    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.335541 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.335514    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"]
Apr 25 00:16:24.351125 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cpg\" (UniqueName: \"kubernetes.io/projected/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kube-api-access-j9cpg\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.351295 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.351295 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351295 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.351295 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351459 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.351459 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351459 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351459 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351647 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.351647 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2pk\" (UniqueName: \"kubernetes.io/projected/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kube-api-access-hb2pk\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351748 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.351793 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.351793 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.351774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452260 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452218    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452260 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452262    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452287    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cpg\" (UniqueName: \"kubernetes.io/projected/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kube-api-access-j9cpg\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452311    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452334    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452354    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452372    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452398    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452424    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452471    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452509 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452497    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452541    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452584    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2pk\" (UniqueName: \"kubernetes.io/projected/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kube-api-access-hb2pk\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452653    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.452993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452712    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452750    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.452993 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.452772    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.453329 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.453010    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.453329 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.453087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.453329 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.453119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.453329 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.453246    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.453507 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.453486    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.454957 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.454934    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.455105 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.455086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.455656 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.455634    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.455747 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.455639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.460502 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.460480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cpg\" (UniqueName: \"kubernetes.io/projected/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kube-api-access-j9cpg\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.460675 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.460657    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2pk\" (UniqueName: \"kubernetes.io/projected/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kube-api-access-hb2pk\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.618343 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.618294    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:24.639629 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.639558    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:16:24.766629 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.766579    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"]
Apr 25 00:16:24.769046 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:16:24.767825    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbc3a7c_9a23_427c_a617_9ceef5b9140c.slice/crio-10eebc01ab1de149a7ea8dcfa09dc56da15c12e7efa1b4c337f8f1ab580d6055 WatchSource:0}: Error finding container 10eebc01ab1de149a7ea8dcfa09dc56da15c12e7efa1b4c337f8f1ab580d6055: Status 404 returned error can't find the container with id 10eebc01ab1de149a7ea8dcfa09dc56da15c12e7efa1b4c337f8f1ab580d6055
Apr 25 00:16:24.771988 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.771962    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 25 00:16:24.791313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.791291    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"]
Apr 25 00:16:24.793198 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:16:24.793170    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab12ec4_a2f7_455b_b8bb_7cc81f6aba9e.slice/crio-c950882f005126c69627bc55cedd926f6fb48649decb6fcd1337e970be0cc9ac WatchSource:0}: Error finding container c950882f005126c69627bc55cedd926f6fb48649decb6fcd1337e970be0cc9ac: Status 404 returned error can't find the container with id c950882f005126c69627bc55cedd926f6fb48649decb6fcd1337e970be0cc9ac
Apr 25 00:16:24.810359 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.810331    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" event={"ID":"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e","Type":"ContainerStarted","Data":"c950882f005126c69627bc55cedd926f6fb48649decb6fcd1337e970be0cc9ac"}
Apr 25 00:16:24.811456 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:24.811435    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" event={"ID":"bdbc3a7c-9a23-427c-a617-9ceef5b9140c","Type":"ContainerStarted","Data":"10eebc01ab1de149a7ea8dcfa09dc56da15c12e7efa1b4c337f8f1ab580d6055"}
Apr 25 00:16:25.817868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:25.817752    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" event={"ID":"bdbc3a7c-9a23-427c-a617-9ceef5b9140c","Type":"ContainerStarted","Data":"585e61dc66ea0d8f7a69432def00a2fe982e9e60a893711f71f460c2940569dd"}
Apr 25 00:16:25.817868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:25.817815    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:16:25.820645 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:25.820353    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" event={"ID":"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e","Type":"ContainerStarted","Data":"75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0"}
Apr 25 00:16:26.826017 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:26.825973    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" event={"ID":"bdbc3a7c-9a23-427c-a617-9ceef5b9140c","Type":"ContainerStarted","Data":"53a7ec9fadfe9b15bc660c6fb5a20d80847366b0978476f4ab5e455f41553d79"}
Apr 25 00:16:32.755925 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.755897    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7d795bc9b6-7tbfw_2c310a32-fd3c-406e-a302-2525e3803c40/main/0.log"
Apr 25 00:16:32.756284 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.756270    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:16:32.840392 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840351    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c56bx\" (UniqueName: \"kubernetes.io/projected/2c310a32-fd3c-406e-a302-2525e3803c40-kube-api-access-c56bx\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.840562 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840411    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-dshm\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.840562 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840443    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c310a32-fd3c-406e-a302-2525e3803c40-tls-certs\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.840737 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840567    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-model-cache\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.840737 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840703    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-kserve-provision-location\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.840831 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840736    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-tmp-dir\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.840831 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.840772    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-home\") pod \"2c310a32-fd3c-406e-a302-2525e3803c40\" (UID: \"2c310a32-fd3c-406e-a302-2525e3803c40\") "
Apr 25 00:16:32.841478 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.841443    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:16:32.841698 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.841676    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-home" (OuterVolumeSpecName: "home") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:16:32.842906 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.842881    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c310a32-fd3c-406e-a302-2525e3803c40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:16:32.843264 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.843229    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-dshm" (OuterVolumeSpecName: "dshm") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:16:32.843346 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.843313    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c310a32-fd3c-406e-a302-2525e3803c40-kube-api-access-c56bx" (OuterVolumeSpecName: "kube-api-access-c56bx") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "kube-api-access-c56bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:16:32.848019 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.847998    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7d795bc9b6-7tbfw_2c310a32-fd3c-406e-a302-2525e3803c40/main/0.log"
Apr 25 00:16:32.848342 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.848315    2576 generic.go:358] "Generic (PLEG): container finished" podID="2c310a32-fd3c-406e-a302-2525e3803c40" containerID="163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2" exitCode=137
Apr 25 00:16:32.848465 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.848397    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"
Apr 25 00:16:32.848465 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.848444    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" event={"ID":"2c310a32-fd3c-406e-a302-2525e3803c40","Type":"ContainerDied","Data":"163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2"}
Apr 25 00:16:32.848575 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.848481    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw" event={"ID":"2c310a32-fd3c-406e-a302-2525e3803c40","Type":"ContainerDied","Data":"d52af229a3ab22733ad6495d38b6d0de44906a7c03ab23dbf9cb63c7e3db7a7a"}
Apr 25 00:16:32.848575 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.848505    2576 scope.go:117] "RemoveContainer" containerID="163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2"
Apr 25 00:16:32.853803 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.853778    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:16:32.862680 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.862655    2576 scope.go:117] "RemoveContainer" containerID="cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e"
Apr 25 00:16:32.909882 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.909840    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c310a32-fd3c-406e-a302-2525e3803c40" (UID: "2c310a32-fd3c-406e-a302-2525e3803c40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:16:32.934322 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.934301    2576 scope.go:117] "RemoveContainer" containerID="163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2"
Apr 25 00:16:32.934684 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:16:32.934656    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2\": container with ID starting with 163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2 not found: ID does not exist" containerID="163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2"
Apr 25 00:16:32.934785 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.934693    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2"} err="failed to get container status \"163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2\": rpc error: code = NotFound desc = could not find container \"163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2\": container with ID starting with 163a80881a0283f8d66a1b7effe45b688416dce8b6aa66a2c659bd3fb54adfc2 not found: ID does not exist"
Apr 25 00:16:32.934785 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.934714    2576 scope.go:117] "RemoveContainer" containerID="cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e"
Apr 25 00:16:32.934972 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:16:32.934949    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e\": container with ID starting with cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e not found: ID does not exist" containerID="cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e"
Apr 25 00:16:32.935012 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.934978    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e"} err="failed to get container status \"cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e\": rpc error: code = NotFound desc = could not find container \"cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e\": container with ID starting with cf2ea151d00ffc25e1366412982aa56bc6919451f935e6ca05703c3171188b1e not found: ID does not exist"
Apr 25 00:16:32.942175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942153    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56bx\" (UniqueName: \"kubernetes.io/projected/2c310a32-fd3c-406e-a302-2525e3803c40-kube-api-access-c56bx\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:32.942235 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942181    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:32.942235 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942196    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c310a32-fd3c-406e-a302-2525e3803c40-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:32.942235 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942208    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:32.942235 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942221    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:32.942235 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942234    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:32.942393 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:32.942246    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c310a32-fd3c-406e-a302-2525e3803c40-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:16:33.169317 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:33.169279    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"]
Apr 25 00:16:33.173832 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:33.173809    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7d795bc9b6-7tbfw"]
Apr 25 00:16:34.873546 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:34.873510    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" path="/var/lib/kubelet/pods/2c310a32-fd3c-406e-a302-2525e3803c40/volumes"
Apr 25 00:16:37.843514 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:16:37.843484    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:19:00.888370 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:00.888328    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:19:00.895091 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:00.895067    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:19:30.469962 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:30.469916    2576 generic.go:358] "Generic (PLEG): container finished" podID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerID="75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0" exitCode=0
Apr 25 00:19:30.470374 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:30.469966    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" event={"ID":"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e","Type":"ContainerDied","Data":"75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0"}
Apr 25 00:19:31.476208 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:31.476161    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" event={"ID":"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e","Type":"ContainerStarted","Data":"2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92"}
Apr 25 00:19:31.497415 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:31.497352    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podStartSLOduration=187.497333186 podStartE2EDuration="3m7.497333186s" podCreationTimestamp="2026-04-25 00:16:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:19:31.495347263 +0000 UTC m=+1531.120386951" watchObservedRunningTime="2026-04-25 00:19:31.497333186 +0000 UTC m=+1531.122372865"
Apr 25 00:19:34.640560 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:34.640523    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:19:34.641043 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:34.640573    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:19:34.642039 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:34.642012    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:19:44.640147 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:44.640094    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:19:54.640493 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:19:54.640446    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:20:04.640495 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:20:04.640447    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:20:14.640131 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:20:14.640088    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:20:24.641019 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:20:24.640968    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:20:34.639965 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:20:34.639921    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:20:44.640357 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:20:44.640249    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:20:54.640576 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:20:54.640535    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Apr 25 00:21:04.650764 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:21:04.650733    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:21:04.658470 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:21:04.658445    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:24:00.914655 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:24:00.914626    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:24:00.922209 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:24:00.922184    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:27:19.064464 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.064428    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"]
Apr 25 00:27:19.064995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.064863    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="storage-initializer"
Apr 25 00:27:19.064995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.064877    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="storage-initializer"
Apr 25 00:27:19.064995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.064886    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main"
Apr 25 00:27:19.064995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.064893    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main"
Apr 25 00:27:19.064995 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.064961    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="2c310a32-fd3c-406e-a302-2525e3803c40" containerName="main"
Apr 25 00:27:19.068379 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.068353    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.070757 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.070734    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
Apr 25 00:27:19.080025 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.079992    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080157 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrk46\" (UniqueName: \"kubernetes.io/projected/92069f9a-baf9-4558-9e1f-4b41d41615eb-kube-api-access-wrk46\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080157 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080283 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92069f9a-baf9-4558-9e1f-4b41d41615eb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080283 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080283 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080447 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.080583 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.080562    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"]
Apr 25 00:27:19.181358 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181317    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181376    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181420    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrk46\" (UniqueName: \"kubernetes.io/projected/92069f9a-baf9-4558-9e1f-4b41d41615eb-kube-api-access-wrk46\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181449    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181514    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92069f9a-baf9-4558-9e1f-4b41d41615eb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181574 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181555    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181894 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181586    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.181958 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.182015 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.181961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.182015 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.182004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.182267 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.182164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.183785 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.183762    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.184089 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.184071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92069f9a-baf9-4558-9e1f-4b41d41615eb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.191584 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.191552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrk46\" (UniqueName: \"kubernetes.io/projected/92069f9a-baf9-4558-9e1f-4b41d41615eb-kube-api-access-wrk46\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.382781 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.382701    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:27:19.525067 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:27:19.524664    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92069f9a_baf9_4558_9e1f_4b41d41615eb.slice/crio-1508ae59f1cf2f1446c212d1d981c63d0f129811950ab880c1e5dcc290b1671b WatchSource:0}: Error finding container 1508ae59f1cf2f1446c212d1d981c63d0f129811950ab880c1e5dcc290b1671b: Status 404 returned error can't find the container with id 1508ae59f1cf2f1446c212d1d981c63d0f129811950ab880c1e5dcc290b1671b
Apr 25 00:27:19.525067 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.524721    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"]
Apr 25 00:27:19.528295 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:19.528272    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 25 00:27:20.116266 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:20.116226    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" event={"ID":"92069f9a-baf9-4558-9e1f-4b41d41615eb","Type":"ContainerStarted","Data":"ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe"}
Apr 25 00:27:20.116266 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:27:20.116271    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" event={"ID":"92069f9a-baf9-4558-9e1f-4b41d41615eb","Type":"ContainerStarted","Data":"1508ae59f1cf2f1446c212d1d981c63d0f129811950ab880c1e5dcc290b1671b"}
Apr 25 00:29:00.950830 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:00.950797    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:29:00.953214 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:00.953188    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:29:01.481201 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:01.481146    2576 generic.go:358] "Generic (PLEG): container finished" podID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerID="ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe" exitCode=0
Apr 25 00:29:01.481379 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:01.481224    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" event={"ID":"92069f9a-baf9-4558-9e1f-4b41d41615eb","Type":"ContainerDied","Data":"ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe"}
Apr 25 00:29:02.486744 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:02.486707    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" event={"ID":"92069f9a-baf9-4558-9e1f-4b41d41615eb","Type":"ContainerStarted","Data":"f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d"}
Apr 25 00:29:02.510121 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:02.510069    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podStartSLOduration=103.510052445 podStartE2EDuration="1m43.510052445s" podCreationTimestamp="2026-04-25 00:27:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:29:02.507838418 +0000 UTC m=+2102.132878136" watchObservedRunningTime="2026-04-25 00:29:02.510052445 +0000 UTC m=+2102.135092120"
Apr 25 00:29:09.383505 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:09.383467    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:29:09.383505 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:09.383509    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:29:09.385327 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:09.385298    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:29:19.383845 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:19.383791    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:29:29.383814 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:29.383771    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:29:39.383423 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:39.383374    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:29:49.383530 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:49.383433    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:29:59.383138 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:29:59.383085    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:30:09.383616 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:09.383550    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:30:19.383717 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:19.383665    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:30:29.383463 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:29.383409    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Apr 25 00:30:39.393865 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:39.393832    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:30:39.402701 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:39.402659    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:30:44.371721 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:44.371686    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"]
Apr 25 00:30:44.372127 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:44.371972    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main" containerID="cri-o://f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d" gracePeriod=30
Apr 25 00:30:56.489906 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.489868    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 25 00:30:56.495303 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.495272    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.498142 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.498115    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Apr 25 00:30:56.498313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.498286    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-bc7j6\""
Apr 25 00:30:56.508201 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.508173    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 25 00:30:56.602829 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.602791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.602829 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.602830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.603086 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.602927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.603086 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.602948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.603086 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.603013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1608941-b987-4291-8222-c2eaea631638-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.603086 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.603074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.603297 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.603099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tzg7\" (UniqueName: \"kubernetes.io/projected/b1608941-b987-4291-8222-c2eaea631638-kube-api-access-4tzg7\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704036 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.703999    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1608941-b987-4291-8222-c2eaea631638-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704036 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704043    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704265 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704062    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzg7\" (UniqueName: \"kubernetes.io/projected/b1608941-b987-4291-8222-c2eaea631638-kube-api-access-4tzg7\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704265 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704132    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704265 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704166    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704265 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704254    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704289    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704449    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704586 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704586 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704534    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.704768 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.704591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.706635 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.706579    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.706809 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.706793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1608941-b987-4291-8222-c2eaea631638-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.712303 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.712274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzg7\" (UniqueName: \"kubernetes.io/projected/b1608941-b987-4291-8222-c2eaea631638-kube-api-access-4tzg7\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.807032 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.806920    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:30:56.951402 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:56.951369    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 25 00:30:56.953206 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:30:56.953177    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1608941_b987_4291_8222_c2eaea631638.slice/crio-a302387a39811767d2196830f1a4e3eaaad518e0db4b828766bd90a57fa12ed0 WatchSource:0}: Error finding container a302387a39811767d2196830f1a4e3eaaad518e0db4b828766bd90a57fa12ed0: Status 404 returned error can't find the container with id a302387a39811767d2196830f1a4e3eaaad518e0db4b828766bd90a57fa12ed0
Apr 25 00:30:57.913531 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:57.913487    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1608941-b987-4291-8222-c2eaea631638","Type":"ContainerStarted","Data":"2a5da6817e46e136a463b5cf7107470ea3e77131bc8e9be56619db230241b7fd"}
Apr 25 00:30:57.913531 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:30:57.913533    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1608941-b987-4291-8222-c2eaea631638","Type":"ContainerStarted","Data":"a302387a39811767d2196830f1a4e3eaaad518e0db4b828766bd90a57fa12ed0"}
Apr 25 00:31:06.996145 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:06.996083    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"]
Apr 25 00:31:06.996678 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:06.996491    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" containerID="cri-o://585e61dc66ea0d8f7a69432def00a2fe982e9e60a893711f71f460c2940569dd" gracePeriod=30
Apr 25 00:31:06.996678 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:06.996548    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="storage-initializer" containerID="cri-o://53a7ec9fadfe9b15bc660c6fb5a20d80847366b0978476f4ab5e455f41553d79" gracePeriod=30
Apr 25 00:31:06.998270 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:06.998242    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"]
Apr 25 00:31:06.998555 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:06.998523    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main" containerID="cri-o://2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92" gracePeriod=30
Apr 25 00:31:07.830410 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:07.830358    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Apr 25 00:31:07.951465 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:07.951433    2576 generic.go:358] "Generic (PLEG): container finished" podID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerID="585e61dc66ea0d8f7a69432def00a2fe982e9e60a893711f71f460c2940569dd" exitCode=0
Apr 25 00:31:07.951665 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:07.951509    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" event={"ID":"bdbc3a7c-9a23-427c-a617-9ceef5b9140c","Type":"ContainerDied","Data":"585e61dc66ea0d8f7a69432def00a2fe982e9e60a893711f71f460c2940569dd"}
Apr 25 00:31:14.619047 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.619001    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Apr 25 00:31:14.646454 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.646420    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct_92069f9a-baf9-4558-9e1f-4b41d41615eb/main/0.log"
Apr 25 00:31:14.646823 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.646805    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:31:14.777966 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.777926    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-home\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778162 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.777983    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrk46\" (UniqueName: \"kubernetes.io/projected/92069f9a-baf9-4558-9e1f-4b41d41615eb-kube-api-access-wrk46\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778162 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778025    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-kserve-provision-location\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778162 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778081    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-tmp-dir\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778162 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778116    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92069f9a-baf9-4558-9e1f-4b41d41615eb-tls-certs\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778162 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778146    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-model-cache\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778380 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778174    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-dshm\") pod \"92069f9a-baf9-4558-9e1f-4b41d41615eb\" (UID: \"92069f9a-baf9-4558-9e1f-4b41d41615eb\") "
Apr 25 00:31:14.778731 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778696    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:14.778878 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.778761    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-home" (OuterVolumeSpecName: "home") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:14.780530 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.780499    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-dshm" (OuterVolumeSpecName: "dshm") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:14.780719 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.780543    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92069f9a-baf9-4558-9e1f-4b41d41615eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:31:14.780790 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.780772    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92069f9a-baf9-4558-9e1f-4b41d41615eb-kube-api-access-wrk46" (OuterVolumeSpecName: "kube-api-access-wrk46") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "kube-api-access-wrk46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:31:14.796232 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.796191    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:14.835162 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.835098    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92069f9a-baf9-4558-9e1f-4b41d41615eb" (UID: "92069f9a-baf9-4558-9e1f-4b41d41615eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879146    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879174    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879185    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92069f9a-baf9-4558-9e1f-4b41d41615eb-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879195    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879203    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879211    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92069f9a-baf9-4558-9e1f-4b41d41615eb-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.879229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.879218    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrk46\" (UniqueName: \"kubernetes.io/projected/92069f9a-baf9-4558-9e1f-4b41d41615eb-kube-api-access-wrk46\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:14.977219 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.977185    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct_92069f9a-baf9-4558-9e1f-4b41d41615eb/main/0.log"
Apr 25 00:31:14.977579 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.977548    2576 generic.go:358] "Generic (PLEG): container finished" podID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerID="f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d" exitCode=137
Apr 25 00:31:14.977675 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.977639    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"
Apr 25 00:31:14.977726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.977638    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" event={"ID":"92069f9a-baf9-4558-9e1f-4b41d41615eb","Type":"ContainerDied","Data":"f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d"}
Apr 25 00:31:14.977779 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.977746    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct" event={"ID":"92069f9a-baf9-4558-9e1f-4b41d41615eb","Type":"ContainerDied","Data":"1508ae59f1cf2f1446c212d1d981c63d0f129811950ab880c1e5dcc290b1671b"}
Apr 25 00:31:14.977779 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.977771    2576 scope.go:117] "RemoveContainer" containerID="f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d"
Apr 25 00:31:14.986854 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.986831    2576 scope.go:117] "RemoveContainer" containerID="ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe"
Apr 25 00:31:14.995439 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.995411    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"]
Apr 25 00:31:14.998726 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:14.998703    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-9b59467c8-pg4ct"]
Apr 25 00:31:15.050163 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:15.050141    2576 scope.go:117] "RemoveContainer" containerID="f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d"
Apr 25 00:31:15.050512 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:31:15.050485    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d\": container with ID starting with f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d not found: ID does not exist" containerID="f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d"
Apr 25 00:31:15.050566 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:15.050522    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d"} err="failed to get container status \"f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d\": rpc error: code = NotFound desc = could not find container \"f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d\": container with ID starting with f0c20e6f55ef1febe44ea91e2ac2346e382f77075bc0716603c18f5053216f4d not found: ID does not exist"
Apr 25 00:31:15.050566 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:15.050542    2576 scope.go:117] "RemoveContainer" containerID="ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe"
Apr 25 00:31:15.051105 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:31:15.051086    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe\": container with ID starting with ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe not found: ID does not exist" containerID="ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe"
Apr 25 00:31:15.051151 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:15.051112    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe"} err="failed to get container status \"ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe\": rpc error: code = NotFound desc = could not find container \"ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe\": container with ID starting with ad189db37f1a5782ddc21adfaf46a1b3cc19730ce04066894f40e4da981fcdbe not found: ID does not exist"
Apr 25 00:31:16.874080 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:16.874047    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" path="/var/lib/kubelet/pods/92069f9a-baf9-4558-9e1f-4b41d41615eb/volumes"
Apr 25 00:31:17.830576 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:17.830529    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Apr 25 00:31:24.619442 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.619395    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Apr 25 00:31:24.697662 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.697623    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"]
Apr 25 00:31:24.698041 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.698027    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main"
Apr 25 00:31:24.698083 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.698041    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main"
Apr 25 00:31:24.698083 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.698076    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="storage-initializer"
Apr 25 00:31:24.698268 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.698082    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="storage-initializer"
Apr 25 00:31:24.698268 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.698141    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="92069f9a-baf9-4558-9e1f-4b41d41615eb" containerName="main"
Apr 25 00:31:24.704198 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.704178    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.707107 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.707072    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Apr 25 00:31:24.707253 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.707124    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-7j7cl\""
Apr 25 00:31:24.708552 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.708532    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"]
Apr 25 00:31:24.714068 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.714039    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"]
Apr 25 00:31:24.714200 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.714161    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.724650 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.724625    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"]
Apr 25 00:31:24.772679 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.772679 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.772889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8whf8\" (UniqueName: \"kubernetes.io/projected/2aee6993-05b7-4a72-885c-04aa03cbf528-kube-api-access-8whf8\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.772889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.772889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.772889 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.773020 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrq7v\" (UniqueName: \"kubernetes.io/projected/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kube-api-access-vrq7v\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.773020 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.773020 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aee6993-05b7-4a72-885c-04aa03cbf528-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.773020 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.772993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.773144 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.773034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.773144 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.773054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.773144 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.773071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-home\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.773144 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.773093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.873997 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.873911    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.873997 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.873947    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.873997 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.873971    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vrq7v\" (UniqueName: \"kubernetes.io/projected/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kube-api-access-vrq7v\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.873997 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.873999    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874347 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874045    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aee6993-05b7-4a72-885c-04aa03cbf528-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874347 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874101    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.874347 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874149    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.874347 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874284    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.874347 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874323    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-home\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874361    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874455    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874488    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874521    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8whf8\" (UniqueName: \"kubernetes.io/projected/2aee6993-05b7-4a72-885c-04aa03cbf528-kube-api-access-8whf8\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.874641 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874551    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.875064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.875064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.875064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874875    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.875064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.874945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-home\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.875264 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.875207    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.875342 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.875322    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.876705 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.876683    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-dshm\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.876841 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.876826    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.876963 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.876943    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aee6993-05b7-4a72-885c-04aa03cbf528-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.877335 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.877321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:24.882850 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.882819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8whf8\" (UniqueName: \"kubernetes.io/projected/2aee6993-05b7-4a72-885c-04aa03cbf528-kube-api-access-8whf8\") pod \"custom-route-timeout-pd-test-kserve-5c6f476c56-glll2\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:24.883016 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:24.882989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrq7v\" (UniqueName: \"kubernetes.io/projected/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kube-api-access-vrq7v\") pod \"custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:25.015813 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:25.015781    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:25.026235 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:25.026208    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:31:25.163382 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:25.163352    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"]
Apr 25 00:31:25.166095 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:31:25.166061    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aee6993_05b7_4a72_885c_04aa03cbf528.slice/crio-b202e6abb93603f4f739313bd659e29aa5a13456125c3a2fea1ab0ebf3634603 WatchSource:0}: Error finding container b202e6abb93603f4f739313bd659e29aa5a13456125c3a2fea1ab0ebf3634603: Status 404 returned error can't find the container with id b202e6abb93603f4f739313bd659e29aa5a13456125c3a2fea1ab0ebf3634603
Apr 25 00:31:25.185634 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:25.185591    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"]
Apr 25 00:31:25.187386 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:31:25.187359    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d6077c_02b6_4fdc_9679_1626af0f5ac2.slice/crio-d8baae0b5217e147190cd44e7ab44ea9ada11c10e37adaac76d7d0071e012913 WatchSource:0}: Error finding container d8baae0b5217e147190cd44e7ab44ea9ada11c10e37adaac76d7d0071e012913: Status 404 returned error can't find the container with id d8baae0b5217e147190cd44e7ab44ea9ada11c10e37adaac76d7d0071e012913
Apr 25 00:31:26.018223 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:26.018182    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerStarted","Data":"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"}
Apr 25 00:31:26.018735 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:26.018228    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerStarted","Data":"b202e6abb93603f4f739313bd659e29aa5a13456125c3a2fea1ab0ebf3634603"}
Apr 25 00:31:26.018735 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:26.018447    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:26.019828 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:26.019792    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" event={"ID":"08d6077c-02b6-4fdc-9679-1626af0f5ac2","Type":"ContainerStarted","Data":"3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614"}
Apr 25 00:31:26.019828 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:26.019837    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" event={"ID":"08d6077c-02b6-4fdc-9679-1626af0f5ac2","Type":"ContainerStarted","Data":"d8baae0b5217e147190cd44e7ab44ea9ada11c10e37adaac76d7d0071e012913"}
Apr 25 00:31:27.026483 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:27.026446    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerStarted","Data":"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"}
Apr 25 00:31:27.830403 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:27.830354    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Apr 25 00:31:34.618761 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:34.618717    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Apr 25 00:31:34.619146 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:34.618789    2576 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:31:37.063884 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.063854    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5_bdbc3a7c-9a23-427c-a617-9ceef5b9140c/storage-initializer/0.log"
Apr 25 00:31:37.064553 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.064271    2576 generic.go:358] "Generic (PLEG): container finished" podID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerID="53a7ec9fadfe9b15bc660c6fb5a20d80847366b0978476f4ab5e455f41553d79" exitCode=137
Apr 25 00:31:37.064553 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.064328    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" event={"ID":"bdbc3a7c-9a23-427c-a617-9ceef5b9140c","Type":"ContainerDied","Data":"53a7ec9fadfe9b15bc660c6fb5a20d80847366b0978476f4ab5e455f41553d79"}
Apr 25 00:31:37.195009 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.194985    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5_bdbc3a7c-9a23-427c-a617-9ceef5b9140c/storage-initializer/0.log"
Apr 25 00:31:37.195430 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.195416    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:31:37.239229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.239206    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:31:37.306137 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306049    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-model-cache\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306150    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb2pk\" (UniqueName: \"kubernetes.io/projected/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kube-api-access-hb2pk\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306188    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tmp-dir\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306208    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tls-certs\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306254    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-home\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306282    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kserve-provision-location\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306313 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306306    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-dshm\") pod \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\" (UID: \"bdbc3a7c-9a23-427c-a617-9ceef5b9140c\") "
Apr 25 00:31:37.306673 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306368    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.306673 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306481    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.306673 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306575    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-home" (OuterVolumeSpecName: "home") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.306837 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306713    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.306837 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306733    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.306837 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.306750    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.308490 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.308463    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kube-api-access-hb2pk" (OuterVolumeSpecName: "kube-api-access-hb2pk") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "kube-api-access-hb2pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:31:37.308638 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.308540    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-dshm" (OuterVolumeSpecName: "dshm") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.308961 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.308942    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:31:37.360193 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.360148    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdbc3a7c-9a23-427c-a617-9ceef5b9140c" (UID: "bdbc3a7c-9a23-427c-a617-9ceef5b9140c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.407492 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407454    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kserve-provision-location\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.407684 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407524    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-model-cache\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.407684 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407588    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9cpg\" (UniqueName: \"kubernetes.io/projected/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kube-api-access-j9cpg\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.407684 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407645    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tls-certs\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.407684 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407682    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tmp-dir\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.407906 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407731    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-dshm\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.407906 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407852    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.407906 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.407858    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-home\") pod \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\" (UID: \"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e\") "
Apr 25 00:31:37.408350 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.408313    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb2pk\" (UniqueName: \"kubernetes.io/projected/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kube-api-access-hb2pk\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.408350 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.408339    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.408350 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.408358    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.408739 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.408391    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdbc3a7c-9a23-427c-a617-9ceef5b9140c-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.408739 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.408487    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.408739 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.408633    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-home" (OuterVolumeSpecName: "home") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.410053 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.410029    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kube-api-access-j9cpg" (OuterVolumeSpecName: "kube-api-access-j9cpg") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "kube-api-access-j9cpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:31:37.410392 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.410366    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-dshm" (OuterVolumeSpecName: "dshm") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.410392 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.410373    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:31:37.425279 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.425250    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.477780 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.477737    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" (UID: "6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:31:37.509876 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.509837    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9cpg\" (UniqueName: \"kubernetes.io/projected/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kube-api-access-j9cpg\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.509876 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.509872    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.509876 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.509886    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.510110 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.509894    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.510110 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.509903    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:37.510110 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:37.509911    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:31:38.043188 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.043160    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:31:38.069562 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.069526    2576 generic.go:358] "Generic (PLEG): container finished" podID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerID="2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92" exitCode=137
Apr 25 00:31:38.070009 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.069619    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" event={"ID":"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e","Type":"ContainerDied","Data":"2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92"}
Apr 25 00:31:38.070009 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.069631    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"
Apr 25 00:31:38.070009 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.069660    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48" event={"ID":"6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e","Type":"ContainerDied","Data":"c950882f005126c69627bc55cedd926f6fb48649decb6fcd1337e970be0cc9ac"}
Apr 25 00:31:38.070009 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.069681    2576 scope.go:117] "RemoveContainer" containerID="2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92"
Apr 25 00:31:38.071131 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.071109    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5_bdbc3a7c-9a23-427c-a617-9ceef5b9140c/storage-initializer/0.log"
Apr 25 00:31:38.071525 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.071504    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5" event={"ID":"bdbc3a7c-9a23-427c-a617-9ceef5b9140c","Type":"ContainerDied","Data":"10eebc01ab1de149a7ea8dcfa09dc56da15c12e7efa1b4c337f8f1ab580d6055"}
Apr 25 00:31:38.071584 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.071555    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"
Apr 25 00:31:38.079306 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.079290    2576 scope.go:117] "RemoveContainer" containerID="75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0"
Apr 25 00:31:38.089584 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.089567    2576 scope.go:117] "RemoveContainer" containerID="2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92"
Apr 25 00:31:38.089864 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:31:38.089845    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92\": container with ID starting with 2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92 not found: ID does not exist" containerID="2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92"
Apr 25 00:31:38.089909 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.089873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92"} err="failed to get container status \"2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92\": rpc error: code = NotFound desc = could not find container \"2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92\": container with ID starting with 2af7812a4d379fe7c3ba0c9e370bb71fcf77116c7378a74ff15d0d9eac1d6a92 not found: ID does not exist"
Apr 25 00:31:38.089909 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.089899    2576 scope.go:117] "RemoveContainer" containerID="75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0"
Apr 25 00:31:38.090191 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:31:38.090173    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0\": container with ID starting with 75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0 not found: ID does not exist" containerID="75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0"
Apr 25 00:31:38.090229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.090199    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0"} err="failed to get container status \"75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0\": rpc error: code = NotFound desc = could not find container \"75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0\": container with ID starting with 75820beb26a769f18601a1e87975384076839732a46cc14ad0a92b3841e455e0 not found: ID does not exist"
Apr 25 00:31:38.090229 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.090215    2576 scope.go:117] "RemoveContainer" containerID="53a7ec9fadfe9b15bc660c6fb5a20d80847366b0978476f4ab5e455f41553d79"
Apr 25 00:31:38.094764 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.094742    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"]
Apr 25 00:31:38.098564 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.098541    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-76bhn48"]
Apr 25 00:31:38.101670 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.101644    2576 scope.go:117] "RemoveContainer" containerID="585e61dc66ea0d8f7a69432def00a2fe982e9e60a893711f71f460c2940569dd"
Apr 25 00:31:38.124025 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.123992    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"]
Apr 25 00:31:38.128776 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.128750    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-d89f655b9-bzbv5"]
Apr 25 00:31:38.874534 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.874500    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" path="/var/lib/kubelet/pods/6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e/volumes"
Apr 25 00:31:38.874976 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:31:38.874963    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" path="/var/lib/kubelet/pods/bdbc3a7c-9a23-427c-a617-9ceef5b9140c/volumes"
Apr 25 00:32:34.275311 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:34.275267    2576 generic.go:358] "Generic (PLEG): container finished" podID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerID="f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e" exitCode=0
Apr 25 00:32:34.275886 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:34.275329    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerDied","Data":"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"}
Apr 25 00:32:34.276883 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:34.276866    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 25 00:32:35.282338 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:35.282302    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerStarted","Data":"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"}
Apr 25 00:32:35.305038 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:35.304369    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podStartSLOduration=71.304353379 podStartE2EDuration="1m11.304353379s" podCreationTimestamp="2026-04-25 00:31:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:32:35.30263858 +0000 UTC m=+2314.927678253" watchObservedRunningTime="2026-04-25 00:32:35.304353379 +0000 UTC m=+2314.929393054"
Apr 25 00:32:41.305927 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:41.305876    2576 generic.go:358] "Generic (PLEG): container finished" podID="b1608941-b987-4291-8222-c2eaea631638" containerID="2a5da6817e46e136a463b5cf7107470ea3e77131bc8e9be56619db230241b7fd" exitCode=0
Apr 25 00:32:41.306473 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:41.305953    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1608941-b987-4291-8222-c2eaea631638","Type":"ContainerDied","Data":"2a5da6817e46e136a463b5cf7107470ea3e77131bc8e9be56619db230241b7fd"}
Apr 25 00:32:42.312852 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:42.312815    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1608941-b987-4291-8222-c2eaea631638","Type":"ContainerStarted","Data":"acab5b0aec629f69870f39b3d2f3dae423f3cdad620ad9d2c73fcf49a98808ee"}
Apr 25 00:32:42.333952 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:42.333900    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=106.333885582 podStartE2EDuration="1m46.333885582s" podCreationTimestamp="2026-04-25 00:30:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:32:42.331186693 +0000 UTC m=+2321.956226384" watchObservedRunningTime="2026-04-25 00:32:42.333885582 +0000 UTC m=+2321.958925258"
Apr 25 00:32:45.016272 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:45.016185    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:32:45.016272 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:45.016240    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:32:45.016834 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:45.016791    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:32:46.807786 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:46.807747    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:32:46.809485 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:46.809455    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:32:55.017005 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:55.016958    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:32:56.807129 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:56.807087    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:32:56.807630 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:32:56.807457    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:33:05.017157 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:05.017107    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:33:06.808439 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:06.808384    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:33:15.016735 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:15.016686    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:33:16.808342 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:16.808298    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:33:25.017357 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:25.017306    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:33:26.808068 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:26.808009    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:33:35.017293 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:35.017238    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:33:36.807589 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:36.807546    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:33:45.017224 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:45.017179    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:33:46.807508 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:46.807468    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:33:55.017153 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:55.017089    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:33:56.808399 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:33:56.808354    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:34:00.977420 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:00.977388    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:34:00.981727 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:00.981706    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:34:05.016982 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:05.016941    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:34:06.807878 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:06.807831    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:34:15.017183 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:15.017076    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:34:16.807506 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:16.807462    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:34:25.016821 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:25.016781    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:34:26.807465 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:26.807410    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:34:35.016976 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:35.016926    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8001/health\": dial tcp 10.134.0.40:8001: connect: connection refused"
Apr 25 00:34:36.807847 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:36.807804    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:34:45.026353 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:45.026318    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:34:45.044196 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:45.044166    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:34:46.808039 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:46.807993    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused"
Apr 25 00:34:54.817456 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:54.817362    2576 generic.go:358] "Generic (PLEG): container finished" podID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerID="3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614" exitCode=0
Apr 25 00:34:54.817456 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:54.817442    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" event={"ID":"08d6077c-02b6-4fdc-9679-1626af0f5ac2","Type":"ContainerDied","Data":"3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614"}
Apr 25 00:34:55.822443 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:55.822403    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" event={"ID":"08d6077c-02b6-4fdc-9679-1626af0f5ac2","Type":"ContainerStarted","Data":"7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488"}
Apr 25 00:34:55.843922 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:55.843862    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podStartSLOduration=211.843842082 podStartE2EDuration="3m31.843842082s" podCreationTimestamp="2026-04-25 00:31:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:34:55.840810719 +0000 UTC m=+2455.465850395" watchObservedRunningTime="2026-04-25 00:34:55.843842082 +0000 UTC m=+2455.468881758"
Apr 25 00:34:56.818451 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:56.818420    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:34:56.833865 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:34:56.833832    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:35:05.026743 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.026705    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:35:05.027210 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.026758    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:35:05.028542 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.028506    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:35:05.192270 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.192228    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 25 00:35:05.192588 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.192560    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main" containerID="cri-o://acab5b0aec629f69870f39b3d2f3dae423f3cdad620ad9d2c73fcf49a98808ee" gracePeriod=30
Apr 25 00:35:05.866806 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.866770    2576 generic.go:358] "Generic (PLEG): container finished" podID="b1608941-b987-4291-8222-c2eaea631638" containerID="acab5b0aec629f69870f39b3d2f3dae423f3cdad620ad9d2c73fcf49a98808ee" exitCode=0
Apr 25 00:35:05.866961 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.866858    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1608941-b987-4291-8222-c2eaea631638","Type":"ContainerDied","Data":"acab5b0aec629f69870f39b3d2f3dae423f3cdad620ad9d2c73fcf49a98808ee"}
Apr 25 00:35:05.940085 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:05.940061    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:35:06.096497 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096445    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-kserve-provision-location\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.096497 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096496    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-model-cache\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096514    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-tmp-dir\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096572    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-home\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096630    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-dshm\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096659    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1608941-b987-4291-8222-c2eaea631638-tls-certs\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096689    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tzg7\" (UniqueName: \"kubernetes.io/projected/b1608941-b987-4291-8222-c2eaea631638-kube-api-access-4tzg7\") pod \"b1608941-b987-4291-8222-c2eaea631638\" (UID: \"b1608941-b987-4291-8222-c2eaea631638\") "
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.096772    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:35:06.097064 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.097032    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.097441 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.097182    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-home" (OuterVolumeSpecName: "home") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:35:06.099496 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.099467    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1608941-b987-4291-8222-c2eaea631638-kube-api-access-4tzg7" (OuterVolumeSpecName: "kube-api-access-4tzg7") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "kube-api-access-4tzg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:35:06.099737 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.099707    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-dshm" (OuterVolumeSpecName: "dshm") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:35:06.099854 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.099781    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1608941-b987-4291-8222-c2eaea631638-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:35:06.110469 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.110442    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:35:06.163069 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.163020    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1608941-b987-4291-8222-c2eaea631638" (UID: "b1608941-b987-4291-8222-c2eaea631638"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:35:06.198585 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.198541    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.198585 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.198579    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.198585 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.198594    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.198857 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.198633    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1608941-b987-4291-8222-c2eaea631638-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.198857 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.198645    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1608941-b987-4291-8222-c2eaea631638-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.198857 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.198657    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tzg7\" (UniqueName: \"kubernetes.io/projected/b1608941-b987-4291-8222-c2eaea631638-kube-api-access-4tzg7\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:35:06.872187 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.872146    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:35:06.876178 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.876138    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1608941-b987-4291-8222-c2eaea631638","Type":"ContainerDied","Data":"a302387a39811767d2196830f1a4e3eaaad518e0db4b828766bd90a57fa12ed0"}
Apr 25 00:35:06.876368 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.876190    2576 scope.go:117] "RemoveContainer" containerID="acab5b0aec629f69870f39b3d2f3dae423f3cdad620ad9d2c73fcf49a98808ee"
Apr 25 00:35:06.887926 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:06.887901    2576 scope.go:117] "RemoveContainer" containerID="2a5da6817e46e136a463b5cf7107470ea3e77131bc8e9be56619db230241b7fd"
Apr 25 00:35:15.027863 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:15.027816    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:35:25.026743 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:25.026703    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:35:35.027233 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:35.027186    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:35:36.883159 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:36.883113    2576 pod_container_manager_linux.go:217] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podb1608941-b987-4291-8222-c2eaea631638"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb1608941-b987-4291-8222-c2eaea631638] : Timed out while waiting for systemd to remove kubepods-burstable-podb1608941_b987_4291_8222_c2eaea631638.slice"
Apr 25 00:35:36.883159 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:35:36.883155    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable podb1608941-b987-4291-8222-c2eaea631638] : unable to destroy cgroup paths for cgroup [kubepods burstable podb1608941-b987-4291-8222-c2eaea631638] : Timed out while waiting for systemd to remove kubepods-burstable-podb1608941_b987_4291_8222_c2eaea631638.slice" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1608941-b987-4291-8222-c2eaea631638"
Apr 25 00:35:36.982769 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:36.982736    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 25 00:35:37.006497 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:37.006452    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 25 00:35:37.009762 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:37.009731    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 25 00:35:38.874314 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:38.874279    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1608941-b987-4291-8222-c2eaea631638" path="/var/lib/kubelet/pods/b1608941-b987-4291-8222-c2eaea631638/volumes"
Apr 25 00:35:45.026878 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:45.026779    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:35:55.026896 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:35:55.026851    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:36:05.026743 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:05.026699    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:36:15.026968 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:15.026924    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:36:25.028377 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:25.028318    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Apr 25 00:36:35.037837 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:35.037801    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:36:35.045912 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:35.045887    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:36:43.506982 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:43.506942    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"]
Apr 25 00:36:43.507484 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:43.507390    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main" containerID="cri-o://166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506" gracePeriod=30
Apr 25 00:36:43.511384 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:43.511356    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"]
Apr 25 00:36:43.511734 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:36:43.511680    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main" containerID="cri-o://7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488" gracePeriod=30
Apr 25 00:37:13.508402 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:13.508330    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="llm-d-routing-sidecar" containerID="cri-o://b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7" gracePeriod=2
Apr 25 00:37:13.659062 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:37:13.659026    2576 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d6077c_02b6_4fdc_9679_1626af0f5ac2.slice/crio-7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aee6993_05b7_4a72_885c_04aa03cbf528.slice/crio-conmon-166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506.scope\": RecentStats: unable to find data in memory cache]"
Apr 25 00:37:13.946124 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:13.946096    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-5c6f476c56-glll2_2aee6993-05b7-4a72-885c-04aa03cbf528/main/0.log"
Apr 25 00:37:13.946752 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:13.946735    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:37:13.949651 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:13.949633    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:37:14.046004 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.045920    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-dshm\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.046004 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.045959    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-kserve-provision-location\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046004 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.045985    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-tmp-dir\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046019    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-model-cache\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046049    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-model-cache\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046078    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrq7v\" (UniqueName: \"kubernetes.io/projected/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kube-api-access-vrq7v\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046131    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aee6993-05b7-4a72-885c-04aa03cbf528-tls-certs\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046155    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-home\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046180    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tls-certs\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046225    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8whf8\" (UniqueName: \"kubernetes.io/projected/2aee6993-05b7-4a72-885c-04aa03cbf528-kube-api-access-8whf8\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046267    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-home\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046271    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-model-cache" (OuterVolumeSpecName: "model-cache") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.046868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046283    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-model-cache" (OuterVolumeSpecName: "model-cache") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.046868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046295    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-dshm\") pod \"2aee6993-05b7-4a72-885c-04aa03cbf528\" (UID: \"2aee6993-05b7-4a72-885c-04aa03cbf528\") "
Apr 25 00:37:14.046868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046365    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tmp-dir\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.046868 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046417    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kserve-provision-location\") pod \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\" (UID: \"08d6077c-02b6-4fdc-9679-1626af0f5ac2\") "
Apr 25 00:37:14.047084 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.046961    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.047084 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.047001    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-model-cache\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.048349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.048014    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-home" (OuterVolumeSpecName: "home") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.048349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.048286    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-home" (OuterVolumeSpecName: "home") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.049040 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.049007    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:37:14.049195 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.049159    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-dshm" (OuterVolumeSpecName: "dshm") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.049538 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.049506    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-dshm" (OuterVolumeSpecName: "dshm") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.050089 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.050064    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aee6993-05b7-4a72-885c-04aa03cbf528-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 25 00:37:14.050533 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.050507    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aee6993-05b7-4a72-885c-04aa03cbf528-kube-api-access-8whf8" (OuterVolumeSpecName: "kube-api-access-8whf8") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "kube-api-access-8whf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:37:14.051063 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.051033    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kube-api-access-vrq7v" (OuterVolumeSpecName: "kube-api-access-vrq7v") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "kube-api-access-vrq7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 25 00:37:14.060696 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.060671    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.061050 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.061031    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.112266 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.112224    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2aee6993-05b7-4a72-885c-04aa03cbf528" (UID: "2aee6993-05b7-4a72-885c-04aa03cbf528"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.113108 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.113084    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08d6077c-02b6-4fdc-9679-1626af0f5ac2" (UID: "08d6077c-02b6-4fdc-9679-1626af0f5ac2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 25 00:37:14.148340 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148307    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aee6993-05b7-4a72-885c-04aa03cbf528-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148340 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148336    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148349    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tls-certs\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148361    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8whf8\" (UniqueName: \"kubernetes.io/projected/2aee6993-05b7-4a72-885c-04aa03cbf528-kube-api-access-8whf8\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148370    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-home\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148378    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148387    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148396    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148404    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08d6077c-02b6-4fdc-9679-1626af0f5ac2-dshm\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148413    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-kserve-provision-location\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148421    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aee6993-05b7-4a72-885c-04aa03cbf528-tmp-dir\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.148472 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.148436    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrq7v\" (UniqueName: \"kubernetes.io/projected/08d6077c-02b6-4fdc-9679-1626af0f5ac2-kube-api-access-vrq7v\") on node \"ip-10-0-134-82.ec2.internal\" DevicePath \"\""
Apr 25 00:37:14.333938 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.333898    2576 generic.go:358] "Generic (PLEG): container finished" podID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerID="7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488" exitCode=137
Apr 25 00:37:14.334125 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.333962    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"
Apr 25 00:37:14.334125 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.333985    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" event={"ID":"08d6077c-02b6-4fdc-9679-1626af0f5ac2","Type":"ContainerDied","Data":"7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488"}
Apr 25 00:37:14.334125 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.334024    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p" event={"ID":"08d6077c-02b6-4fdc-9679-1626af0f5ac2","Type":"ContainerDied","Data":"d8baae0b5217e147190cd44e7ab44ea9ada11c10e37adaac76d7d0071e012913"}
Apr 25 00:37:14.334125 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.334044    2576 scope.go:117] "RemoveContainer" containerID="7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488"
Apr 25 00:37:14.335588 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.335571    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-5c6f476c56-glll2_2aee6993-05b7-4a72-885c-04aa03cbf528/main/0.log"
Apr 25 00:37:14.336170 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.336147    2576 generic.go:358] "Generic (PLEG): container finished" podID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerID="166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506" exitCode=137
Apr 25 00:37:14.336170 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.336166    2576 generic.go:358] "Generic (PLEG): container finished" podID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerID="b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7" exitCode=0
Apr 25 00:37:14.336349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.336217    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"
Apr 25 00:37:14.336349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.336233    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerDied","Data":"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"}
Apr 25 00:37:14.336349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.336274    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerDied","Data":"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"}
Apr 25 00:37:14.336349 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.336287    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2" event={"ID":"2aee6993-05b7-4a72-885c-04aa03cbf528","Type":"ContainerDied","Data":"b202e6abb93603f4f739313bd659e29aa5a13456125c3a2fea1ab0ebf3634603"}
Apr 25 00:37:14.343412 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.343370    2576 scope.go:117] "RemoveContainer" containerID="3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614"
Apr 25 00:37:14.354939 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.354803    2576 scope.go:117] "RemoveContainer" containerID="7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488"
Apr 25 00:37:14.355194 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:37:14.355165    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488\": container with ID starting with 7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488 not found: ID does not exist" containerID="7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488"
Apr 25 00:37:14.355338 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.355204    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488"} err="failed to get container status \"7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488\": rpc error: code = NotFound desc = could not find container \"7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488\": container with ID starting with 7b273e94df637a9a8361c5a4c0ee243c4334da5fcfd9d82b1bcf2653e5d0a488 not found: ID does not exist"
Apr 25 00:37:14.355338 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.355229    2576 scope.go:117] "RemoveContainer" containerID="3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614"
Apr 25 00:37:14.355515 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:37:14.355498    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614\": container with ID starting with 3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614 not found: ID does not exist" containerID="3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614"
Apr 25 00:37:14.355561 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.355522    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614"} err="failed to get container status \"3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614\": rpc error: code = NotFound desc = could not find container \"3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614\": container with ID starting with 3ac8a1e50e502edb99b0cdea210c7faf9c949969647e1d3f65425078464ea614 not found: ID does not exist"
Apr 25 00:37:14.355561 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.355540    2576 scope.go:117] "RemoveContainer" containerID="166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"
Apr 25 00:37:14.356852 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.356828    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"]
Apr 25 00:37:14.368546 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.368522    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-d695c97ff-rm46p"]
Apr 25 00:37:14.370243 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.370228    2576 scope.go:117] "RemoveContainer" containerID="f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"
Apr 25 00:37:14.379060 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.378966    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"]
Apr 25 00:37:14.382141 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.382111    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5c6f476c56-glll2"]
Apr 25 00:37:14.441711 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.441681    2576 scope.go:117] "RemoveContainer" containerID="b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"
Apr 25 00:37:14.449675 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.449651    2576 scope.go:117] "RemoveContainer" containerID="166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"
Apr 25 00:37:14.449968 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:37:14.449943    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506\": container with ID starting with 166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506 not found: ID does not exist" containerID="166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"
Apr 25 00:37:14.450022 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.449977    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"} err="failed to get container status \"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506\": rpc error: code = NotFound desc = could not find container \"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506\": container with ID starting with 166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506 not found: ID does not exist"
Apr 25 00:37:14.450022 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.449999    2576 scope.go:117] "RemoveContainer" containerID="f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"
Apr 25 00:37:14.450266 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:37:14.450248    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e\": container with ID starting with f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e not found: ID does not exist" containerID="f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"
Apr 25 00:37:14.450323 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450271    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"} err="failed to get container status \"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e\": rpc error: code = NotFound desc = could not find container \"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e\": container with ID starting with f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e not found: ID does not exist"
Apr 25 00:37:14.450323 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450295    2576 scope.go:117] "RemoveContainer" containerID="b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"
Apr 25 00:37:14.450536 ip-10-0-134-82 kubenswrapper[2576]: E0425 00:37:14.450521    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7\": container with ID starting with b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7 not found: ID does not exist" containerID="b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"
Apr 25 00:37:14.450577 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450541    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"} err="failed to get container status \"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7\": rpc error: code = NotFound desc = could not find container \"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7\": container with ID starting with b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7 not found: ID does not exist"
Apr 25 00:37:14.450577 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450555    2576 scope.go:117] "RemoveContainer" containerID="166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"
Apr 25 00:37:14.450797 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450778    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506"} err="failed to get container status \"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506\": rpc error: code = NotFound desc = could not find container \"166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506\": container with ID starting with 166846d507475bab93fee4af86b9b5861d531f464232c5b515e82e92a2362506 not found: ID does not exist"
Apr 25 00:37:14.450842 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450797    2576 scope.go:117] "RemoveContainer" containerID="f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"
Apr 25 00:37:14.450998 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450984    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e"} err="failed to get container status \"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e\": rpc error: code = NotFound desc = could not find container \"f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e\": container with ID starting with f09721c6a46f11c1952452c0348e9371165487102d0b02ab5ac8de6d3e073b5e not found: ID does not exist"
Apr 25 00:37:14.451041 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.450998    2576 scope.go:117] "RemoveContainer" containerID="b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"
Apr 25 00:37:14.451215 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.451197    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7"} err="failed to get container status \"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7\": rpc error: code = NotFound desc = could not find container \"b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7\": container with ID starting with b51e499a11ad8c732812844c5d48d33435e02070021fa9ca7913537d21bc7ab7 not found: ID does not exist"
Apr 25 00:37:14.874498 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.874462    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" path="/var/lib/kubelet/pods/08d6077c-02b6-4fdc-9679-1626af0f5ac2/volumes"
Apr 25 00:37:14.874920 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:37:14.874906    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" path="/var/lib/kubelet/pods/2aee6993-05b7-4a72-885c-04aa03cbf528/volumes"
Apr 25 00:39:01.002386 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:39:01.002312    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:39:01.008244 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:39:01.008220    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:44:01.027898 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:44:01.027867    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:44:01.034308 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:44:01.034267    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:49:01.054274 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:49:01.054160    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:49:01.060072 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:49:01.060052    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:50:24.875053 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:24.875022    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:25.987165 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:25.987134    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:27.090187 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:27.090157    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:28.163785 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:28.163759    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:29.261553 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:29.261526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:30.339215 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:30.339176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:31.434966 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:31.434931    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:32.540816 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:32.540789    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:33.622099 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:33.622062    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:34.726711 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:34.726671    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:35.844306 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:35.844261    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:37.008237 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:37.008202    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:38.102925 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:38.102887    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:39.294249 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:39.294218    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-65cb4d7898-bdr5z_13974f39-01b9-4b0e-928e-7aefc9ff940f/storage-initializer/0.log"
Apr 25 00:50:42.948751 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:42.948715    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-f9rbg_257f3d2b-2299-441e-b58b-dfb37bfe9713/manager/0.log"
Apr 25 00:50:42.958652 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:42.958624    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-m86g7_e8f5d4e9-a22b-40b8-9683-5a7f58b1056f/limitador/0.log"
Apr 25 00:50:43.879137 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:43.879046    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-f9rbg_257f3d2b-2299-441e-b58b-dfb37bfe9713/manager/0.log"
Apr 25 00:50:43.888563 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:43.888535    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-m86g7_e8f5d4e9-a22b-40b8-9683-5a7f58b1056f/limitador/0.log"
Apr 25 00:50:44.792571 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:44.792532    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-f9rbg_257f3d2b-2299-441e-b58b-dfb37bfe9713/manager/0.log"
Apr 25 00:50:44.800861 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:44.800842    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-m86g7_e8f5d4e9-a22b-40b8-9683-5a7f58b1056f/limitador/0.log"
Apr 25 00:50:45.684966 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:45.684931    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-f9rbg_257f3d2b-2299-441e-b58b-dfb37bfe9713/manager/0.log"
Apr 25 00:50:45.693310 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:45.693283    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-m86g7_e8f5d4e9-a22b-40b8-9683-5a7f58b1056f/limitador/0.log"
Apr 25 00:50:46.586616 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:46.586542    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-f9rbg_257f3d2b-2299-441e-b58b-dfb37bfe9713/manager/0.log"
Apr 25 00:50:46.598636 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:46.598611    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-m86g7_e8f5d4e9-a22b-40b8-9683-5a7f58b1056f/limitador/0.log"
Apr 25 00:50:51.638389 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:51.638355    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9m6vr_762b6285-4d40-434c-b48c-cc06271acb74/global-pull-secret-syncer/0.log"
Apr 25 00:50:51.776744 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:51.776712    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hrx4r_ec46b4ce-5ee5-4d29-8d14-8e3071852804/konnectivity-agent/0.log"
Apr 25 00:50:51.898462 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:51.898383    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-82.ec2.internal_20602652b465b1663ffa2d71aa406f85/haproxy/0.log"
Apr 25 00:50:55.976863 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:55.976824    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-f9rbg_257f3d2b-2299-441e-b58b-dfb37bfe9713/manager/0.log"
Apr 25 00:50:55.996675 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:55.996646    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-m86g7_e8f5d4e9-a22b-40b8-9683-5a7f58b1056f/limitador/0.log"
Apr 25 00:50:56.992457 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:56.992345    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/alertmanager/0.log"
Apr 25 00:50:57.028519 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.028489    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/config-reloader/0.log"
Apr 25 00:50:57.053930 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.053902    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/kube-rbac-proxy-web/0.log"
Apr 25 00:50:57.073519 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.073490    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/kube-rbac-proxy/0.log"
Apr 25 00:50:57.096439 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.096405    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/kube-rbac-proxy-metric/0.log"
Apr 25 00:50:57.133207 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.133177    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/prom-label-proxy/0.log"
Apr 25 00:50:57.162870 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.162836    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2d0c0a8-9e99-47ca-bb69-c7fbfa816c54/init-config-reloader/0.log"
Apr 25 00:50:57.232593 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.232552    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-qnfzz_54c60e27-d809-4a9b-9f8c-fa8c15b9bf34/kube-state-metrics/0.log"
Apr 25 00:50:57.250950 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.250873    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-qnfzz_54c60e27-d809-4a9b-9f8c-fa8c15b9bf34/kube-rbac-proxy-main/0.log"
Apr 25 00:50:57.270193 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.270166    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-qnfzz_54c60e27-d809-4a9b-9f8c-fa8c15b9bf34/kube-rbac-proxy-self/0.log"
Apr 25 00:50:57.294320 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.294284    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5776cf486c-bmk6q_ad93192c-534f-4ab1-ae0d-091859d832b2/metrics-server/0.log"
Apr 25 00:50:57.318537 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.318506    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7dccd58f55-n8z9m_ff43f34c-be72-4604-b7a1-be1ab3ba9ce4/monitoring-plugin/0.log"
Apr 25 00:50:57.421321 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.421288    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zqb26_778fb23e-9606-41ce-a9a7-ba09b3ecbfb3/node-exporter/0.log"
Apr 25 00:50:57.438915 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.438879    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zqb26_778fb23e-9606-41ce-a9a7-ba09b3ecbfb3/kube-rbac-proxy/0.log"
Apr 25 00:50:57.462047 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.462018    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zqb26_778fb23e-9606-41ce-a9a7-ba09b3ecbfb3/init-textfile/0.log"
Apr 25 00:50:57.598727 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.598694    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-59prq_db99497f-4c14-45c0-bfc0-7d8e6c70cee8/kube-rbac-proxy-main/0.log"
Apr 25 00:50:57.646394 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.646365    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-59prq_db99497f-4c14-45c0-bfc0-7d8e6c70cee8/kube-rbac-proxy-self/0.log"
Apr 25 00:50:57.689063 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.689020    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-59prq_db99497f-4c14-45c0-bfc0-7d8e6c70cee8/openshift-state-metrics/0.log"
Apr 25 00:50:57.745664 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.745635    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/prometheus/0.log"
Apr 25 00:50:57.766888 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.766865    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/config-reloader/0.log"
Apr 25 00:50:57.790413 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.790385    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/thanos-sidecar/0.log"
Apr 25 00:50:57.810408 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.810385    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/kube-rbac-proxy-web/0.log"
Apr 25 00:50:57.835215 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.835190    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/kube-rbac-proxy/0.log"
Apr 25 00:50:57.855954 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.855880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/kube-rbac-proxy-thanos/0.log"
Apr 25 00:50:57.882382 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.882354    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_45ba6210-0d2c-4431-9f81-cdfdff431c6f/init-config-reloader/0.log"
Apr 25 00:50:57.925998 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.925967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5676c8c784-2st66_8fe51d1e-6b60-455a-aace-77aea96906e0/prometheus-operator/0.log"
Apr 25 00:50:57.952927 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:57.952896    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5676c8c784-2st66_8fe51d1e-6b60-455a-aace-77aea96906e0/kube-rbac-proxy/0.log"
Apr 25 00:50:58.013652 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.013623    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fb4d7d57c-ws2z5_8cb439d7-f947-477e-a49c-be8a9e3b3c32/telemeter-client/0.log"
Apr 25 00:50:58.038478 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.038445    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fb4d7d57c-ws2z5_8cb439d7-f947-477e-a49c-be8a9e3b3c32/reload/0.log"
Apr 25 00:50:58.066250 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.066226    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fb4d7d57c-ws2z5_8cb439d7-f947-477e-a49c-be8a9e3b3c32/kube-rbac-proxy/0.log"
Apr 25 00:50:58.098420 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.098394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbb5b4cc9-d9h9q_6e9a1e5d-0ba8-41db-bb50-31e22229e9b8/thanos-query/0.log"
Apr 25 00:50:58.118292 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.118219    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbb5b4cc9-d9h9q_6e9a1e5d-0ba8-41db-bb50-31e22229e9b8/kube-rbac-proxy-web/0.log"
Apr 25 00:50:58.140263 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.140229    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbb5b4cc9-d9h9q_6e9a1e5d-0ba8-41db-bb50-31e22229e9b8/kube-rbac-proxy/0.log"
Apr 25 00:50:58.168092 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.168067    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbb5b4cc9-d9h9q_6e9a1e5d-0ba8-41db-bb50-31e22229e9b8/prom-label-proxy/0.log"
Apr 25 00:50:58.189294 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.189258    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbb5b4cc9-d9h9q_6e9a1e5d-0ba8-41db-bb50-31e22229e9b8/kube-rbac-proxy-rules/0.log"
Apr 25 00:50:58.208586 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:50:58.208559    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbb5b4cc9-d9h9q_6e9a1e5d-0ba8-41db-bb50-31e22229e9b8/kube-rbac-proxy-metrics/0.log"
Apr 25 00:51:00.504110 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504068    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"]
Apr 25 00:51:00.504642 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504622    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504643    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504661    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504669    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504681    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="storage-initializer"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504688    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="storage-initializer"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504708    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="storage-initializer"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504717    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="storage-initializer"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504729    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main"
Apr 25 00:51:00.504733 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504737    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504746    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504756    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504785    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504796    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="llm-d-routing-sidecar"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504804    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="llm-d-routing-sidecar"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504814    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504822    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504836    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504844    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504854    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504862    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504961    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="08d6077c-02b6-4fdc-9679-1626af0f5ac2" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504975    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="6ab12ec4-a2f7-455b-b8bb-7cc81f6aba9e" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504986    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.504995    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="storage-initializer"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.505005    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1608941-b987-4291-8222-c2eaea631638" containerName="main"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.505017    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="2aee6993-05b7-4a72-885c-04aa03cbf528" containerName="llm-d-routing-sidecar"
Apr 25 00:51:00.505175 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.505027    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="bdbc3a7c-9a23-427c-a617-9ceef5b9140c" containerName="llm-d-routing-sidecar"
Apr 25 00:51:00.508681 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.508660    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.510997 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.510975    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtbkd\"/\"default-dockercfg-svszf\""
Apr 25 00:51:00.511813 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.511794    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtbkd\"/\"kube-root-ca.crt\""
Apr 25 00:51:00.511911 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.511830    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtbkd\"/\"openshift-service-ca.crt\""
Apr 25 00:51:00.517209 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.517187    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"]
Apr 25 00:51:00.660111 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.660072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-lib-modules\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.660289 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.660116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-sys\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.660289 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.660157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-proc\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.660289 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.660190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw56x\" (UniqueName: \"kubernetes.io/projected/ecad7714-2e8d-425f-90a2-d2d23eee4acd-kube-api-access-pw56x\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.660289 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.660232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-podres\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761280 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761171    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-lib-modules\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761280 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761229    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-sys\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761280 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761277    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-proc\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761543 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761308    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-sys\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761543 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761326    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pw56x\" (UniqueName: \"kubernetes.io/projected/ecad7714-2e8d-425f-90a2-d2d23eee4acd-kube-api-access-pw56x\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761543 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-lib-modules\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761543 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-proc\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761543 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761404    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-podres\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.761543 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.761519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ecad7714-2e8d-425f-90a2-d2d23eee4acd-podres\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.768615 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.768580    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtbkd\"/\"kube-root-ca.crt\""
Apr 25 00:51:00.779348 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.779330    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtbkd\"/\"openshift-service-ca.crt\""
Apr 25 00:51:00.790810 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.790783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw56x\" (UniqueName: \"kubernetes.io/projected/ecad7714-2e8d-425f-90a2-d2d23eee4acd-kube-api-access-pw56x\") pod \"perf-node-gather-daemonset-gfvgv\" (UID: \"ecad7714-2e8d-425f-90a2-d2d23eee4acd\") " pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.821688 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.821658    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtbkd\"/\"default-dockercfg-svszf\""
Apr 25 00:51:00.830283 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.830261    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:00.956533 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.956506    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"]
Apr 25 00:51:00.959329 ip-10-0-134-82 kubenswrapper[2576]: W0425 00:51:00.959287    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podecad7714_2e8d_425f_90a2_d2d23eee4acd.slice/crio-853463e9797cea254131f9d0258e1b278fc509d380daba80b4ab1be0da773e38 WatchSource:0}: Error finding container 853463e9797cea254131f9d0258e1b278fc509d380daba80b4ab1be0da773e38: Status 404 returned error can't find the container with id 853463e9797cea254131f9d0258e1b278fc509d380daba80b4ab1be0da773e38
Apr 25 00:51:00.960938 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:00.960918    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 25 00:51:01.183968 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.183932    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv" event={"ID":"ecad7714-2e8d-425f-90a2-d2d23eee4acd","Type":"ContainerStarted","Data":"501c739195cdbd1e05b4ac109717d5458f00372fb7dbced4977658adf839d855"}
Apr 25 00:51:01.183968 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.183968    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv" event={"ID":"ecad7714-2e8d-425f-90a2-d2d23eee4acd","Type":"ContainerStarted","Data":"853463e9797cea254131f9d0258e1b278fc509d380daba80b4ab1be0da773e38"}
Apr 25 00:51:01.184186 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.184054    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:01.199546 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.199490    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv" podStartSLOduration=1.199476054 podStartE2EDuration="1.199476054s" podCreationTimestamp="2026-04-25 00:51:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-25 00:51:01.198843915 +0000 UTC m=+3420.823883604" watchObservedRunningTime="2026-04-25 00:51:01.199476054 +0000 UTC m=+3420.824515730"
Apr 25 00:51:01.711951 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.711922    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l5jkt_67d24f30-2b79-44ee-bf51-10b4444a5035/dns/0.log"
Apr 25 00:51:01.731078 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.731043    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l5jkt_67d24f30-2b79-44ee-bf51-10b4444a5035/kube-rbac-proxy/0.log"
Apr 25 00:51:01.756190 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:01.756160    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jdgg_70fb4ead-406c-4a68-9fdd-bd950fced4a6/dns-node-resolver/0.log"
Apr 25 00:51:02.320200 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:02.320174    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d6fnc_0cba9d02-9ab8-40ba-a254-3ee4430edef7/node-ca/0.log"
Apr 25 00:51:03.685573 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:03.685542    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8dxqr_11529be1-47c8-4118-a866-36eb412239c8/serve-healthcheck-canary/0.log"
Apr 25 00:51:04.245802 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:04.245777    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86bbr_89554b8a-f951-4924-8546-b51462423669/kube-rbac-proxy/0.log"
Apr 25 00:51:04.270001 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:04.269970    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86bbr_89554b8a-f951-4924-8546-b51462423669/exporter/0.log"
Apr 25 00:51:04.291485 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:04.291454    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86bbr_89554b8a-f951-4924-8546-b51462423669/extractor/0.log"
Apr 25 00:51:06.855708 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:06.855666    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58fbc56fdc-ql9rn_3153a201-d81f-49d7-b7b9-7da26ac414fc/manager/0.log"
Apr 25 00:51:07.198021 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:07.197947    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dtbkd/perf-node-gather-daemonset-gfvgv"
Apr 25 00:51:13.818260 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.818234    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/kube-multus-additional-cni-plugins/0.log"
Apr 25 00:51:13.838399 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.838362    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/egress-router-binary-copy/0.log"
Apr 25 00:51:13.857369 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.857340    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/cni-plugins/0.log"
Apr 25 00:51:13.878710 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.878682    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/bond-cni-plugin/0.log"
Apr 25 00:51:13.898103 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.898068    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/routeoverride-cni/0.log"
Apr 25 00:51:13.917293 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.917256    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/whereabouts-cni-bincopy/0.log"
Apr 25 00:51:13.936198 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:13.936174    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-655v2_e44bd199-2f3d-41c4-9044-8635f902289e/whereabouts-cni/0.log"
Apr 25 00:51:14.292282 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:14.292251    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jrgm6_fcc79ac7-41f1-46b6-b86b-a35706a53212/kube-multus/0.log"
Apr 25 00:51:14.436128 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:14.436093    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tcvmr_5b867a9a-b211-492e-a7c9-6081e587a486/network-metrics-daemon/0.log"
Apr 25 00:51:14.452508 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:14.452481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tcvmr_5b867a9a-b211-492e-a7c9-6081e587a486/kube-rbac-proxy/0.log"
Apr 25 00:51:15.241379 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.241347    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-controller/0.log"
Apr 25 00:51:15.256389 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.256348    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/0.log"
Apr 25 00:51:15.288618 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.288555    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovn-acl-logging/1.log"
Apr 25 00:51:15.306834 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.306805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/kube-rbac-proxy-node/0.log"
Apr 25 00:51:15.330012 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.329984    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/kube-rbac-proxy-ovn-metrics/0.log"
Apr 25 00:51:15.348953 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.348916    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/northd/0.log"
Apr 25 00:51:15.368063 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.368040    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/nbdb/0.log"
Apr 25 00:51:15.388662 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.388637    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/sbdb/0.log"
Apr 25 00:51:15.569816 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:15.569730    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9z2gl_253f240c-2da1-4b34-ab6f-9696ac273b3c/ovnkube-controller/0.log"
Apr 25 00:51:17.443948 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:17.443917    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9ltqc_b54c5cf9-9b7c-4090-a9e4-e1aadbb64dc3/network-check-target-container/0.log"
Apr 25 00:51:18.464863 ip-10-0-134-82 kubenswrapper[2576]: I0425 00:51:18.464831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pkr4p_b1923eed-7e79-485f-8f7a-a5c7799e6a40/iptables-alerter/0.log"