Jul 01 14:36:43.040312 ip-10-0-128-59 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 14:36:43.040326 ip-10-0-128-59 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 14:36:43.040335 ip-10-0-128-59 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 14:36:43.040661 ip-10-0-128-59 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 14:36:53.052907 ip-10-0-128-59 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 14:36:53.052927 ip-10-0-128-59 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 515f69e3909345cf972f6079efa02f2d --
Jul 01 14:38:41.353577 ip-10-0-128-59 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 14:38:41.790054 ip-10-0-128-59 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 14:38:41.790054 ip-10-0-128-59 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 14:38:41.790054 ip-10-0-128-59 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 14:38:41.790054 ip-10-0-128-59 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 14:38:41.790706 ip-10-0-128-59 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 14:38:41.790706 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.790127    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 14:38:41.794728 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794712    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:38:41.794728 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794727    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794731    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794734    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794738    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794740    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794743    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794746    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794749    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794752    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794755    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794758    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794762    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794766    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794778    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794781    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794784    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794787    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794790    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794793    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:38:41.794796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794795    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794798    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794801    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794804    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794808    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794811    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794813    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794816    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794819    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794821    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794824    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794827    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794829    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794832    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794834    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794838    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794840    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794843    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794845    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:38:41.795245 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794849    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794852    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794855    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794859    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794861    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794864    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794866    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794869    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794871    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794873    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794876    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794879    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794881    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794883    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794886    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794890    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794892    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794895    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794897    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794900    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794902    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:38:41.795894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794905    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794907    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794910    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794912    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794915    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794917    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794920    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794923    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794925    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794928    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794930    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794933    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794935    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794938    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794940    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794945    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794949    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794951    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794954    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794957    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:38:41.796425 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794959    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794962    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794964    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794967    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794969    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794972    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794975    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794978    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794980    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794983    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794985    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794988    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794990    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794993    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794995    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.794998    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.795000    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.795003    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:38:41.796933 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.795006    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:38:41.798084 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798065    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798084    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798090    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798095    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798099    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798103    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798106    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798112    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798117    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798120    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:38:41.798121 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798125    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798129    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798133    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798137    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798141    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798145    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798149    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798153    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798156    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798160    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798163    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798167    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798171    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798175    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798179    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798182    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798185    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798189    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798193    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:38:41.798386 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798197    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798202    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798206    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798211    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798215    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798219    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798222    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798226    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798229    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798233    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798239    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798245    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798250    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798254    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798257    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798261    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798265    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798269    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798273    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:38:41.798867 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798277    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798281    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798284    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798288    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798292    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798296    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798312    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798316    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798320    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798324    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798328    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798333    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798337    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798340    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798344    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798348    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798353    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798357    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798361    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798364    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:38:41.799349 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798369    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798373    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798379    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798384    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798389    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798393    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798397    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798400    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798406    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798410    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798413    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798416    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798420    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798423    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798427    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798433    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798436    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798440    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798443    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:38:41.799837 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798447    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798451    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798454    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798458    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798462    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798466    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798469    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798473    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798477    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798481    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798484    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.798488    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799022    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799029    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799033    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799037    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799041    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799045    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799049    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799052    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:38:41.800297 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799056    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799060    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799064    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799068    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799073    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799077    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799081    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799085    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799090    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799094    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799098    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799102    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799105    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799109    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799113    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799117    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799121    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799124    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799129    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799132    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:38:41.800796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799136    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799140    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799144    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799147    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799151    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799156    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799160    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799164    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799168    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799171    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799175    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799178    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799181    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799185    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799189    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799193    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799197    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799200    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799204    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:38:41.801273 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799207    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799211    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799215    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799220    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799224    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799227    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799230    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799234    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799237    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799241    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799245    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799248    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799252    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799255    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799259    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799262    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799268    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799272    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799278    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799283    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:38:41.801759 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799287    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799291    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799296    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799314    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799320    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799324    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799328    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799331    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799335    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799339    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799343    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799349    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799354    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799358    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799362    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799367    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799371    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799375    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799379    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:38:41.802275 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799383    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799387    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799391    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799394    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799398    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799401    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799405    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799409    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799412    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799416    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799419    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799423    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.799426    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799516    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799525    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799535    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799541    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799547    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799552    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799558    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799564    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 14:38:41.802767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799569    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799573    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799577    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799582    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799587    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799591    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799595    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799599    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799604    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799609    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799615    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799618    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799623    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799627    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799632    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799637    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799641    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799645    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799649    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799653    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799658    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799662    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799667    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799671    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 14:38:41.803254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799676    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799681    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799686    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799690    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799694    2579 flags.go:64] FLAG: --enable-server="true"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799698    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799704    2579 flags.go:64] FLAG: --event-burst="100"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799709    2579 flags.go:64] FLAG: --event-qps="50"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799713    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799717    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799721    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799727    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799731    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799735    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799740    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799744    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799749    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799753    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799758    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799762    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799766    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799770    2579 flags.go:64] FLAG: --feature-gates=""
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799775    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799780    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799784    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 14:38:41.803861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799789    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799792    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799796    2579 flags.go:64] FLAG: --help="false"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799801    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-59.ec2.internal"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799805    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799810    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799814    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799819    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799824    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799828    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799832    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799837    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799841    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799846    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799850    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799854    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799859    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799863    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799867    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799871    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799875    2579 flags.go:64] FLAG: --lock-file=""
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799879    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799883    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799887    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 14:38:41.804596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799895    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799899    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799903    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799907    2579 flags.go:64] FLAG: --logging-format="text"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799911    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799916    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799920    2579 flags.go:64] FLAG: --manifest-url=""
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799924    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799930    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799934    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799940    2579 flags.go:64] FLAG: --max-pods="110"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799944    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799951    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799955    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799959    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799964    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799969    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799972    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799982    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799987    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799991    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.799996    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800000    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 14:38:41.805189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800007    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800011    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800016    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800019    2579 flags.go:64] FLAG: --port="10250"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800024    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800027    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06908ee4f754aa714"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800032    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800036    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800041    2579 flags.go:64] FLAG: --register-node="true"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800045    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800050    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800055    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800059    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800063    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800068    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800072    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800076    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800080    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800084    2579 flags.go:64] FLAG: --runonce="false"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800088    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800092    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800099    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800103    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800107    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800111    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800116    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 14:38:41.805815 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800121    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800125    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800129    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800133    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800136    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800141    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800145    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800149    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800156    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800160    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800164    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800169    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800173    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800177    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800180    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800185    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800189    2579 flags.go:64] FLAG: --v="2"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800195    2579 flags.go:64] FLAG: --version="false"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800200    2579 flags.go:64] FLAG: --vmodule=""
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800206    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800210    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800386    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800392    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800399    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:38:41.806464 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800403    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800407    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800412    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800417    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800420    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800424    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800428    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800432    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800436    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800440    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800443    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800447    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800450    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800454    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800458    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800463    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800466    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800470    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800473    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800477    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:38:41.807048 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800481    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800485    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800489    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800492    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800495    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800499    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800503    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800508    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800512    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800515    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800518    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800523    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800528    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800531    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800535    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800540    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800544    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800547    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800551    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800555    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:38:41.807614 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800559    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800563    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800566    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800569    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800573    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800577    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800581    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800585    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800588    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800592    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800596    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800599    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800603    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800607    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800610    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800616    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800620    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800625    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800629    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:38:41.808115 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800633    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800637    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800641    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800646    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800651    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800655    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800659    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800663    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800675    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800679    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800682    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800686    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800690    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800694    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800697    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800701    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800705    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800709    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800713    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:38:41.808582 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800717    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800720    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800725    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800730    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800734    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800740    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800745    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800748    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800752    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800757    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800761    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800765    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800768    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800771    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800775    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800779    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800783    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:38:41.809049 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.800789    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.800795    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.808260    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.808276    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.808297    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.808316    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808373    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808378    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808381    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808383    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808387    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808389    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808392    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808395    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:38:41.809475 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808398    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808401    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808404    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808406    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808409    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808412    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808414    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808417    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808420    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808422    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808425    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808427    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808431    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808435    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808438    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808441    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808444    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808447    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808450    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:38:41.809835 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808453    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808456    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808459    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808462    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808465    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808468    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808470    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808473    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808475    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808478    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808481    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808483    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808486    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808503    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808507    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808511    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808515    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808518    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808521    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808524    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:38:41.810293 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808527    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808529    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808532    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808535    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808537    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808539    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808543    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808545    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808548    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808551    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808554    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808557    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808559    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808562    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808565    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808567    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808570    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808573    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808575    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:38:41.810796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808578    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808580    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808582    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808585    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808588    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808590    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808593    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808596    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808599    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808601    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808604    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808606    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808609    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808611    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808614    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808616    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808619    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808621    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808624    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808626    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:38:41.811290 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808629    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808631    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808634    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808637    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808640    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808642    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808645    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808648    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808651    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808653    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808657    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808661    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808663    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.808669    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808777    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 14:38:41.811811 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808782    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808785    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808787    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808790    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808793    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808795    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808798    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808801    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808803    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808806    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808808    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808812    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808815    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808819    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808821    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808824    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808827    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808830    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808832    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 14:38:41.812327 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808835    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808837    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808842    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808845    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808848    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808852    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808855    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808858    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808861    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808864    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808867    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808869    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808873    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808877    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808879    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808882    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808884    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808887    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808889    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 14:38:41.812804 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808892    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808894    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808897    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808899    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808902    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808904    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808906    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808909    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808911    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808914    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808917    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808919    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808921    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808924    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808927    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808929    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808932    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808936    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808939    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808942    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 14:38:41.813257 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808944    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808947    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808950    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808952    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808955    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808957    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808960    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808963    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808965    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808968    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808970    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808973    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808975    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808978    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808980    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808983    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808985    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808988    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808990    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808993    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 14:38:41.813834 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808995    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.808998    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809001    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809004    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809006    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809008    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809011    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809013    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809016    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809019    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809022    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809024    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809026    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809029    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809031    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809034    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809036    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809039    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809041    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 14:38:41.814315 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:41.809044    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 14:38:41.814783 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.809049    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 14:38:41.814783 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.809150    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 14:38:41.814783 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.811687    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 14:38:41.814783 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.814705    2579 server.go:1019] "Starting client certificate rotation"
Jul 01 14:38:41.815007 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.814829    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 14:38:41.815007 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.814882    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 14:38:41.817402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.817340    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 14:38:41.818922 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.818900    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 14:38:41.829946 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.829930    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 01 14:38:41.831922 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.831908    2579 log.go:25] "Validated CRI v1 image API"
Jul 01 14:38:41.832871 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.832858    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 14:38:41.836313 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.836279    2579 fs.go:135] Filesystem UUIDs: map[0bc77456-c78c-407c-b6c0-513b851cb5b1:/dev/nvme0n1p3 2b5e4dd6-58d7-4b81-b2ef-2b163abcb393:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 01 14:38:41.836377 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.836298    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 14:38:41.843289 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.843177    2579 manager.go:216] Machine: {Timestamp:2026-07-01 14:38:41.841699215 +0000 UTC m=+0.367983210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099949 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b4aa1730a240229ba053848a9e10 SystemUUID:ec24b4aa-1730-a240-229b-a053848a9e10 BootID:515f69e3-9093-45cf-972f-6079efa02f2d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2b:39:98:da:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2b:39:98:da:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:0e:b9:55:5f:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 14:38:41.843289 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.843277    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 14:38:41.843427 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.843374    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 14:38:41.846365 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.846336    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 14:38:41.846540 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.846372    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-59.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 14:38:41.847192 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.847181    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 14:38:41.847221 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.847198    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 14:38:41.847221 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.847207    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 14:38:41.848112 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.848101    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 14:38:41.853004 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.852987    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 14:38:41.860437 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.860424    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 14:38:41.863918 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.863906    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 14:38:41.864038 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864028    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 14:38:41.864080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864059    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 14:38:41.864080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864069    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 14:38:41.864080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864080    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 14:38:41.864181 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864090    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 01 14:38:41.864181 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864099    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 14:38:41.864490 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864466    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 14:38:41.864490 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.864485    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 14:38:41.866865 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.866849    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 14:38:41.868289 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.868275    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 14:38:41.868363 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.868322    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 14:38:41.869019 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869009    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 14:38:41.869053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869025    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 14:38:41.869053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869031    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 14:38:41.869053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869036    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 14:38:41.869053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869048    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869056    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869124    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869130    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869136    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869142    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869150    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 14:38:41.869163 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869159    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 14:38:41.869912 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869902    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 14:38:41.869951 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.869914    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 14:38:41.873348 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.873331    2579 server.go:1268] "Started kubelet"
Jul 01 14:38:41.873700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.873499    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 14:38:41.873700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.873512    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 14:38:41.873832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.873709    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 14:38:41.874204 ip-10-0-128-59 systemd[1]: Started Kubernetes Kubelet.
Jul 01 14:38:41.875644 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.875334    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 14:38:41.877383 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.876327    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 14:38:41.877383 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.877257    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2bwx6"
Jul 01 14:38:41.877596 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.877553    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 14:38:41.877771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.877743    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-59.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 14:38:41.877836 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.877772    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 14:38:41.880673 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.880651    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 14:38:41.883468 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.882061    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-59.ec2.internal.18be314fef5c7b54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-59.ec2.internal,UID:ip-10-0-128-59.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-59.ec2.internal,},FirstTimestamp:2026-07-01 14:38:41.87328802 +0000 UTC m=+0.399572015,LastTimestamp:2026-07-01 14:38:41.87328802 +0000 UTC m=+0.399572015,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-59.ec2.internal,}"
Jul 01 14:38:41.884007 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.883996    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 14:38:41.884231 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.884212    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 14:38:41.884298 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.884244    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:41.884707 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.884692    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 14:38:41.884778 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.884712    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 14:38:41.884778 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.884692    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 14:38:41.884908 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.884885    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 14:38:41.884908 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.884898    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 14:38:41.885143 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885126    2579 factory.go:55] Registering systemd factory
Jul 01 14:38:41.885143 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885143    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 01 14:38:41.885453 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885441    2579 factory.go:153] Registering CRI-O factory
Jul 01 14:38:41.885453 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885455    2579 factory.go:223] Registration of the crio container factory successfully
Jul 01 14:38:41.885581 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885495    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 14:38:41.885581 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885511    2579 factory.go:103] Registering Raw factory
Jul 01 14:38:41.885581 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885521    2579 manager.go:1196] Started watching for new ooms in manager
Jul 01 14:38:41.885708 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885674    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2bwx6"
Jul 01 14:38:41.885958 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.885943    2579 manager.go:349] Starting recovery of all containers
Jul 01 14:38:41.887330 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.887287    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 14:38:41.898782 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.898759    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:38:41.899793 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.899776    2579 manager.go:354] Recovery completed
Jul 01 14:38:41.904518 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.904506    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:41.906366 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906348    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:41.906435 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906390    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:41.906435 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906409    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:41.906945 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906933    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 14:38:41.906945 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906944    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 14:38:41.907022 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906951    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:41.907022 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.906959    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 14:38:41.907665 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.907648    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:41.907767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.907679    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:41.907767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.907697    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:41.908068 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.908054    2579 policy_none.go:49] "None policy: Start"
Jul 01 14:38:41.908068 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.908070    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 14:38:41.908176 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.908081    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 14:38:41.909007 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.908993    2579 policy_none.go:47] "Start"
Jul 01 14:38:41.950050 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.949908    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.951173    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.951207    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.951242    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.951294    2579 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953065    2579 manager.go:337] "Starting Device Plugin manager"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.953092    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953109    2579 server.go:87] "Starting device plugin registration server"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953405    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953420    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953557    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953684    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953692    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:41.953901    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.954331    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 14:38:41.958230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:41.954384    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.052087 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.051979    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"]
Jul 01 14:38:42.052087 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.052073    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:42.053181 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.053166    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:42.053238 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.053197    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:42.053238 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.053211    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:42.053605 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.053590    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:42.054287 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054270    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:42.054390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054324    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:42.054390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054337    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:42.054390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054363    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.054652 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054639    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:42.054811 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.054865 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.054824    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:42.055356 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.055340    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:42.055451 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.055361    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:42.055451 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.055371    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:42.055451 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.055397    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:42.055451 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.055437    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:42.055571 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.055458    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:42.056520 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.056507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.056563 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.056532    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 14:38:42.057158 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.057146    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 14:38:42.057224 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.057166    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 14:38:42.057224 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.057175    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID"
Jul 01 14:38:42.064289 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.064275    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.064361 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.064312    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-59.ec2.internal\": node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.081086 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.081067    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-59.ec2.internal\" not found" node="ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.084216 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.084193    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.085756 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.085737    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-59.ec2.internal\" not found" node="ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.085947 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.085925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8758b1976ce38078ba9ae8f6eae9ad37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"8758b1976ce38078ba9ae8f6eae9ad37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.085985 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.085957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/371354312e904fda882e14f06ac6f309-config\") pod \"kube-apiserver-proxy-ip-10-0-128-59.ec2.internal\" (UID: \"371354312e904fda882e14f06ac6f309\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.086018 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.085988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8758b1976ce38078ba9ae8f6eae9ad37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"8758b1976ce38078ba9ae8f6eae9ad37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.185159 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.185122    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.186296 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.186278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/371354312e904fda882e14f06ac6f309-config\") pod \"kube-apiserver-proxy-ip-10-0-128-59.ec2.internal\" (UID: \"371354312e904fda882e14f06ac6f309\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.186376 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.186347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8758b1976ce38078ba9ae8f6eae9ad37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"8758b1976ce38078ba9ae8f6eae9ad37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.186420 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.186381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/371354312e904fda882e14f06ac6f309-config\") pod \"kube-apiserver-proxy-ip-10-0-128-59.ec2.internal\" (UID: \"371354312e904fda882e14f06ac6f309\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.186420 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.186393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8758b1976ce38078ba9ae8f6eae9ad37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"8758b1976ce38078ba9ae8f6eae9ad37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.186496 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.186415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8758b1976ce38078ba9ae8f6eae9ad37-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"8758b1976ce38078ba9ae8f6eae9ad37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.186496 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.186439    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8758b1976ce38078ba9ae8f6eae9ad37-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"8758b1976ce38078ba9ae8f6eae9ad37\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.285764 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.285729    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.385490 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.385395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.386600 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.386579    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.388787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.388768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:42.489709 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.489669    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.591675 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.591646    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.692791 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.692700    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.793810 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.793775    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.813391 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.813365    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 14:38:42.813526 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.813507    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 14:38:42.813573 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.813549    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 14:38:42.892291 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.892257    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 14:33:42 +0000 UTC" deadline="2028-04-08 19:28:56.820406476 +0000 UTC"
Jul 01 14:38:42.892291 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:42.892287    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15532h50m13.928123491s"
Jul 01 14:38:42.894459 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.894438    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:42.995208 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:42.995119    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:43.046138 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.046106    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:38:43.095916 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:43.095887    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found"
Jul 01 14:38:43.111920 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.111891    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:38:43.157655 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:43.157617    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8758b1976ce38078ba9ae8f6eae9ad37.slice/crio-ca34d5783afa4da96f6471e86d356550a3bfcfbf43c5f17a939d19518508ccdf WatchSource:0}: Error finding container ca34d5783afa4da96f6471e86d356550a3bfcfbf43c5f17a939d19518508ccdf: Status 404 returned error can't find the container with id ca34d5783afa4da96f6471e86d356550a3bfcfbf43c5f17a939d19518508ccdf
Jul 01 14:38:43.158001 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:43.157976    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371354312e904fda882e14f06ac6f309.slice/crio-394dad4dd9ae0351cc7898213547c704c31f22b33c529c02ebbca9a5afb819b6 WatchSource:0}: Error finding container 394dad4dd9ae0351cc7898213547c704c31f22b33c529c02ebbca9a5afb819b6: Status 404 returned error can't find the container with id 394dad4dd9ae0351cc7898213547c704c31f22b33c529c02ebbca9a5afb819b6
Jul 01 14:38:43.162714 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.162698    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 14:38:43.184356 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.184333    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:43.197881 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.197860    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 14:38:43.199395 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.199382    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"
Jul 01 14:38:43.211886 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.211864    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 14:38:43.644908 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.644878    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:38:43.868221 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.868183    2579 apiserver.go:52] "Watching apiserver"
Jul 01 14:38:43.875473 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.875437    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 14:38:43.875877 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.875852    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-vwzh5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal","openshift-multus/multus-additional-cni-plugins-bz8l6","openshift-multus/multus-swpjb","openshift-network-diagnostics/network-check-target-ljxh4","openshift-network-operator/iptables-alerter-hj4w7","openshift-ovn-kubernetes/ovnkube-node-mpdnb","kube-system/global-pull-secret-syncer-xl6ft","kube-system/konnectivity-agent-wdmbn","kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl","openshift-cluster-node-tuning-operator/tuned-6ch5j","openshift-dns/node-resolver-9p5nd","openshift-image-registry/node-ca-8bwfq"]
Jul 01 14:38:43.878223 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.878200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.879470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.879449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:43.880808 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.880788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.881889 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.881858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7wsgd\""
Jul 01 14:38:43.882034 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.881882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 14:38:43.882034 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.881993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.882175 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.881999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 14:38:43.882175 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.882162    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s55xx\""
Jul 01 14:38:43.882282 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.882186    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.882282 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.882231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 14:38:43.882577 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.882562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.882832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.882806    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.882935 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.882839    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 14:38:43.884237 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.883532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:43.884237 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.883599    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.884237 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:43.883650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:43.884237 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.883957    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.884237 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.884104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 14:38:43.884237 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.884155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.885474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.885753    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.885778    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bnn4l\""
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.886466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.886591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.886853    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.886865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rx86t\""
Jul 01 14:38:43.887700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.886921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:43.888296 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.888278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:43.889620 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.889598    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z8pmj\""
Jul 01 14:38:43.889753 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.889676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:43.889812 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:43.889761    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:43.890309 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.890295    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 14:38:43.890470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.890309    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.890470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.890299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.890668 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.890653    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 14:38:43.890794 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.890765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 14:38:43.891065 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.891041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:43.891065 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.891064    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qmbnt\""
Jul 01 14:38:43.893119 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.893101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:43.893224 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:43.893152    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:43.893362 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.893344    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.893723 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.893699    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qwhh8\""
Jul 01 14:38:43.893823 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.893751    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.893960 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.893945    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 14:38:43.894617 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.894601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:43.896048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.896030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:43.896653 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.896639    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 14:38:43.896946 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.896929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.897047 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.897033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.897415 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.897400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tqx42\""
Jul 01 14:38:43.899079 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899065    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f57tt\""
Jul 01 14:38:43.899297 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-cni-multus\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899397 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb2984d-8776-4605-8c15-d0cc4167c2ef-host-slash\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:43.899397 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-systemd\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899397 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-conf-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-etc-kubernetes\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899408    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:38:43.899513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwrt\" (UniqueName: \"kubernetes.io/projected/6f9e4965-e925-4022-a963-8a03bc3b2c3c-kube-api-access-8vwrt\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-system-cni-dir\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.899628 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.899628 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-cni-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899628 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-k8s-cni-cncf-io\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899720 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-netns\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899720 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-kubelet\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899720 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-daemon-config\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899720 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-systemd-units\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899844 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-run-netns\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899844 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899844 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-ovnkube-config\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899844 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.899844 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-cnibin\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.899995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 14:38:43.899995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb2984d-8776-4605-8c15-d0cc4167c2ef-iptables-alerter-script\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:43.899995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-etc-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d360fed-4183-4b87-b1b1-63d8c7964670-ovn-node-metrics-cert\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.899995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.899983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-ovnkube-script-lib\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-os-release\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.900142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-var-lib-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-ovn\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-node-log\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-cni-netd\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cnibin\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-os-release\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9e4965-e925-4022-a963-8a03bc3b2c3c-cni-binary-copy\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-cni-bin\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-env-overrides\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-hostroot\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-multus-certs\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-kubelet\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900367 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-slash\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-log-socket\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtzh\" (UniqueName: \"kubernetes.io/projected/1d360fed-4183-4b87-b1b1-63d8c7964670-kube-api-access-rmtzh\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxrb\" (UniqueName: \"kubernetes.io/projected/35978eac-71cf-4911-a661-c713ab08588d-kube-api-access-vgxrb\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kj78\" (UniqueName: \"kubernetes.io/projected/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-kube-api-access-4kj78\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677kz\" (UniqueName: \"kubernetes.io/projected/4cb2984d-8776-4605-8c15-d0cc4167c2ef-kube-api-access-677kz\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-run-ovn-kubernetes\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35978eac-71cf-4911-a661-c713ab08588d-hosts-file\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35978eac-71cf-4911-a661-c713ab08588d-tmp-dir\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-system-cni-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-socket-dir-parent\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.900704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.900670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-cni-bin\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:43.904786 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.904765    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 14:38:43.934687 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.934665    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jmnsq"
Jul 01 14:38:43.944704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.944682    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jmnsq"
Jul 01 14:38:43.957740 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.957689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" event={"ID":"8758b1976ce38078ba9ae8f6eae9ad37","Type":"ContainerStarted","Data":"ca34d5783afa4da96f6471e86d356550a3bfcfbf43c5f17a939d19518508ccdf"}
Jul 01 14:38:43.958614 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.958591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" event={"ID":"371354312e904fda882e14f06ac6f309","Type":"ContainerStarted","Data":"394dad4dd9ae0351cc7898213547c704c31f22b33c529c02ebbca9a5afb819b6"}
Jul 01 14:38:43.986158 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:43.986120    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 14:38:44.000975 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.000931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-kubelet\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.000975 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.000977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxrb\" (UniqueName: \"kubernetes.io/projected/35978eac-71cf-4911-a661-c713ab08588d-kube-api-access-vgxrb\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.001227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.001227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001072    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-kubelet\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-cni-bin\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.001227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-socket-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.001227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c02426cb-5b55-4f73-ab60-16df7c4dcc2a-agent-certs\") pod \"konnectivity-agent-wdmbn\" (UID: \"c02426cb-5b55-4f73-ab60-16df7c4dcc2a\") " pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.001468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwrt\" (UniqueName: \"kubernetes.io/projected/6f9e4965-e925-4022-a963-8a03bc3b2c3c-kube-api-access-8vwrt\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.001468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001259    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-cni-bin\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.001468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-run-ovn-kubernetes\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001404    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-run-ovn-kubernetes\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35978eac-71cf-4911-a661-c713ab08588d-hosts-file\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.001468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35978eac-71cf-4911-a661-c713ab08588d-tmp-dir\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-registration-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-systemd\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ztw\" (UniqueName: \"kubernetes.io/projected/e079a600-38ba-410b-949c-40bf6ef35e6f-kube-api-access-q2ztw\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001538    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35978eac-71cf-4911-a661-c713ab08588d-hosts-file\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-lib-modules\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-system-cni-dir\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001566    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-systemd\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001615    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-system-cni-dir\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-k8s-cni-cncf-io\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001664    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-systemd-units\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001630    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-k8s-cni-cncf-io\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001708    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-systemd-units\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.001729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c48eacc-cd11-4893-94ef-d8368f954658-kubelet-config\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-os-release\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-systemd\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-host\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35978eac-71cf-4911-a661-c713ab08588d-tmp-dir\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cnibin\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-os-release\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cnibin\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001912    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9e4965-e925-4022-a963-8a03bc3b2c3c-cni-binary-copy\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-os-release\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf54j\" (UniqueName: \"kubernetes.io/projected/12300cae-4145-449e-9bac-bf1b507ab3da-kube-api-access-rf54j\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.001910    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-os-release\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-modprobe-d\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-run\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.002299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-cni-bin\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-multus-certs\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-tuned\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-cni-bin\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002151    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002162    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-multus-certs\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-slash\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-log-socket\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtzh\" (UniqueName: \"kubernetes.io/projected/1d360fed-4183-4b87-b1b1-63d8c7964670-kube-api-access-rmtzh\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002237    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-slash\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kj78\" (UniqueName: \"kubernetes.io/projected/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-kube-api-access-4kj78\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002251    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-log-socket\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-system-cni-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-socket-dir-parent\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-var-lib-kubelet\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.003064 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-conf-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-etc-kubernetes\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677kz\" (UniqueName: \"kubernetes.io/projected/4cb2984d-8776-4605-8c15-d0cc4167c2ef-kube-api-access-677kz\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-conf-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9e4965-e925-4022-a963-8a03bc3b2c3c-cni-binary-copy\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002528    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-etc-kubernetes\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002536    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-socket-dir-parent\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002553    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-system-cni-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-cni-multus\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002593    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-cni-multus\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-sys-fs\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rck\" (UniqueName: \"kubernetes.io/projected/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-kube-api-access-77rck\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb2984d-8776-4605-8c15-d0cc4167c2ef-host-slash\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c48eacc-cd11-4893-94ef-d8368f954658-dbus\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002761    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cni-binary-copy\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.003897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e079a600-38ba-410b-949c-40bf6ef35e6f-host\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysctl-conf\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002818    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb2984d-8776-4605-8c15-d0cc4167c2ef-host-slash\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkd9\" (UniqueName: \"kubernetes.io/projected/c0bbda80-5433-4a27-bfc3-a894363c3548-kube-api-access-9jkd9\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-cni-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-netns\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-kubelet\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-daemon-config\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c02426cb-5b55-4f73-ab60-16df7c4dcc2a-konnectivity-ca\") pod \"konnectivity-agent-wdmbn\" (UID: \"c02426cb-5b55-4f73-ab60-16df7c4dcc2a\") " pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e079a600-38ba-410b-949c-40bf6ef35e6f-serviceca\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002985    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-cni-dir\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003004    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-var-lib-kubelet\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.002995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-run-netns\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003033    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-host-run-netns\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-ovnkube-config\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003070    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-run-netns\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.004671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-cnibin\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003128    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-cnibin\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-device-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysconfig\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-sys\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb2984d-8776-4605-8c15-d0cc4167c2ef-iptables-alerter-script\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-etc-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d360fed-4183-4b87-b1b1-63d8c7964670-ovn-node-metrics-cert\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-ovnkube-script-lib\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-kubernetes\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysctl-d\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-etc-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-var-lib-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003467    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003469    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9e4965-e925-4022-a963-8a03bc3b2c3c-multus-daemon-config\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.005432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-ovn\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003505    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-var-lib-openvswitch\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-node-log\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-run-ovn\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-cni-netd\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003619    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-node-log\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0bbda80-5433-4a27-bfc3-a894363c3548-tmp\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003667    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-ovnkube-config\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-env-overrides\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003707    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d360fed-4183-4b87-b1b1-63d8c7964670-host-cni-netd\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-hostroot\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003789    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9e4965-e925-4022-a963-8a03bc3b2c3c-hostroot\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003879    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-ovnkube-script-lib\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003885    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.003909    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb2984d-8776-4605-8c15-d0cc4167c2ef-iptables-alerter-script\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.005962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.004095    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d360fed-4183-4b87-b1b1-63d8c7964670-env-overrides\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.007331 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.007314    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d360fed-4183-4b87-b1b1-63d8c7964670-ovn-node-metrics-cert\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.010504 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.010476    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:38:44.010653 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.010509    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:38:44.010653 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.010526    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:44.010653 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.010615    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:38:44.510579807 +0000 UTC m=+3.036863813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:44.011333 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.011306    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxrb\" (UniqueName: \"kubernetes.io/projected/35978eac-71cf-4911-a661-c713ab08588d-kube-api-access-vgxrb\") pod \"node-resolver-9p5nd\" (UID: \"35978eac-71cf-4911-a661-c713ab08588d\") " pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.011467 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.011445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwrt\" (UniqueName: \"kubernetes.io/projected/6f9e4965-e925-4022-a963-8a03bc3b2c3c-kube-api-access-8vwrt\") pod \"multus-swpjb\" (UID: \"6f9e4965-e925-4022-a963-8a03bc3b2c3c\") " pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.012115 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.012097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtzh\" (UniqueName: \"kubernetes.io/projected/1d360fed-4183-4b87-b1b1-63d8c7964670-kube-api-access-rmtzh\") pod \"ovnkube-node-mpdnb\" (UID: \"1d360fed-4183-4b87-b1b1-63d8c7964670\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.012808 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.012792    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-677kz\" (UniqueName: \"kubernetes.io/projected/4cb2984d-8776-4605-8c15-d0cc4167c2ef-kube-api-access-677kz\") pod \"iptables-alerter-hj4w7\" (UID: \"4cb2984d-8776-4605-8c15-d0cc4167c2ef\") " pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.012858 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.012817    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kj78\" (UniqueName: \"kubernetes.io/projected/3e44db6a-a5cd-430e-9f12-fb6ce04488ba-kube-api-access-4kj78\") pod \"multus-additional-cni-plugins-bz8l6\" (UID: \"3e44db6a-a5cd-430e-9f12-fb6ce04488ba\") " pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.104314 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-kubernetes\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysctl-d\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0bbda80-5433-4a27-bfc3-a894363c3548-tmp\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104472    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysctl-d\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-socket-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c02426cb-5b55-4f73-ab60-16df7c4dcc2a-agent-certs\") pod \"konnectivity-agent-wdmbn\" (UID: \"c02426cb-5b55-4f73-ab60-16df7c4dcc2a\") " pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.104532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-registration-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ztw\" (UniqueName: \"kubernetes.io/projected/e079a600-38ba-410b-949c-40bf6ef35e6f-kube-api-access-q2ztw\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-lib-modules\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c48eacc-cd11-4893-94ef-d8368f954658-kubelet-config\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104602    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-registration-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-systemd\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-kubernetes\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-host\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104650    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-socket-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104665    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c48eacc-cd11-4893-94ef-d8368f954658-kubelet-config\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf54j\" (UniqueName: \"kubernetes.io/projected/12300cae-4145-449e-9bac-bf1b507ab3da-kube-api-access-rf54j\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104694    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-systemd\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104698    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-host\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-modprobe-d\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-run\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104762    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-lib-modules\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-tuned\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-var-lib-kubelet\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.105032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-run\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-modprobe-d\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-sys-fs\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rck\" (UniqueName: \"kubernetes.io/projected/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-kube-api-access-77rck\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104900    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-var-lib-kubelet\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c48eacc-cd11-4893-94ef-d8368f954658-dbus\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104966    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-etc-selinux\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-sys-fs\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.104980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e079a600-38ba-410b-949c-40bf6ef35e6f-host\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysctl-conf\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.105050    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkd9\" (UniqueName: \"kubernetes.io/projected/c0bbda80-5433-4a27-bfc3-a894363c3548-kube-api-access-9jkd9\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105089    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e079a600-38ba-410b-949c-40bf6ef35e6f-host\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c02426cb-5b55-4f73-ab60-16df7c4dcc2a-konnectivity-ca\") pod \"konnectivity-agent-wdmbn\" (UID: \"c02426cb-5b55-4f73-ab60-16df7c4dcc2a\") " pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e079a600-38ba-410b-949c-40bf6ef35e6f-serviceca\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.106082 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.105166    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:38:44.605144 +0000 UTC m=+3.131428007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105236    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysctl-conf\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-device-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysconfig\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-sys\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105376    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-device-dir\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105406    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-sys\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105456    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-sysconfig\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c48eacc-cd11-4893-94ef-d8368f954658-dbus\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.105476    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.105538    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:38:44.605523437 +0000 UTC m=+3.131807436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105623    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e079a600-38ba-410b-949c-40bf6ef35e6f-serviceca\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.106970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.105678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c02426cb-5b55-4f73-ab60-16df7c4dcc2a-konnectivity-ca\") pod \"konnectivity-agent-wdmbn\" (UID: \"c02426cb-5b55-4f73-ab60-16df7c4dcc2a\") " pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.107470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.107109    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c02426cb-5b55-4f73-ab60-16df7c4dcc2a-agent-certs\") pod \"konnectivity-agent-wdmbn\" (UID: \"c02426cb-5b55-4f73-ab60-16df7c4dcc2a\") " pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.107470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.107335    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0bbda80-5433-4a27-bfc3-a894363c3548-tmp\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.107470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.107381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c0bbda80-5433-4a27-bfc3-a894363c3548-etc-tuned\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.118102 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.118069    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf54j\" (UniqueName: \"kubernetes.io/projected/12300cae-4145-449e-9bac-bf1b507ab3da-kube-api-access-rf54j\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:44.118254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.118069    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rck\" (UniqueName: \"kubernetes.io/projected/a26da1d3-b884-4104-9ec4-c0e1bcf3c333-kube-api-access-77rck\") pod \"aws-ebs-csi-driver-node-p7xxl\" (UID: \"a26da1d3-b884-4104-9ec4-c0e1bcf3c333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.118254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.118208    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ztw\" (UniqueName: \"kubernetes.io/projected/e079a600-38ba-410b-949c-40bf6ef35e6f-kube-api-access-q2ztw\") pod \"node-ca-8bwfq\" (UID: \"e079a600-38ba-410b-949c-40bf6ef35e6f\") " pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.118430 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.118406    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkd9\" (UniqueName: \"kubernetes.io/projected/c0bbda80-5433-4a27-bfc3-a894363c3548-kube-api-access-9jkd9\") pod \"tuned-6ch5j\" (UID: \"c0bbda80-5433-4a27-bfc3-a894363c3548\") " pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.192812 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.192728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:38:44.199156 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.199136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9p5nd"
Jul 01 14:38:44.201845 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.201818    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d360fed_4183_4b87_b1b1_63d8c7964670.slice/crio-12fcb16028897e915db125a83a48e3855452c8de9465a319c5cc577d7c55b221 WatchSource:0}: Error finding container 12fcb16028897e915db125a83a48e3855452c8de9465a319c5cc577d7c55b221: Status 404 returned error can't find the container with id 12fcb16028897e915db125a83a48e3855452c8de9465a319c5cc577d7c55b221
Jul 01 14:38:44.206090 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.206055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bz8l6"
Jul 01 14:38:44.208796 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.208772    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35978eac_71cf_4911_a661_c713ab08588d.slice/crio-9922c222c8ab9acc43d83baff3ca72ef0323fea51deb6c35e3eb20dd6d30bc8d WatchSource:0}: Error finding container 9922c222c8ab9acc43d83baff3ca72ef0323fea51deb6c35e3eb20dd6d30bc8d: Status 404 returned error can't find the container with id 9922c222c8ab9acc43d83baff3ca72ef0323fea51deb6c35e3eb20dd6d30bc8d
Jul 01 14:38:44.211389 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.211333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swpjb"
Jul 01 14:38:44.215787 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.215763    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e44db6a_a5cd_430e_9f12_fb6ce04488ba.slice/crio-88c7a9d46e2759002e721ebf8ae6343733306a98547b2df08debe6f1edcf5381 WatchSource:0}: Error finding container 88c7a9d46e2759002e721ebf8ae6343733306a98547b2df08debe6f1edcf5381: Status 404 returned error can't find the container with id 88c7a9d46e2759002e721ebf8ae6343733306a98547b2df08debe6f1edcf5381
Jul 01 14:38:44.216905 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.216881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hj4w7"
Jul 01 14:38:44.222305 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.222275    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9e4965_e925_4022_a963_8a03bc3b2c3c.slice/crio-eb6ac7909a200c91b11dd14fc040bdd223b8521b84071f2667d6fd049bdb3853 WatchSource:0}: Error finding container eb6ac7909a200c91b11dd14fc040bdd223b8521b84071f2667d6fd049bdb3853: Status 404 returned error can't find the container with id eb6ac7909a200c91b11dd14fc040bdd223b8521b84071f2667d6fd049bdb3853
Jul 01 14:38:44.223528 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.223498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:38:44.228703 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.228681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bwfq"
Jul 01 14:38:44.230491 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.230461    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb2984d_8776_4605_8c15_d0cc4167c2ef.slice/crio-98c32c6058a10b2d197c4f8b68cc1a3bde98ae5a26f67073a122e385ec8240b3 WatchSource:0}: Error finding container 98c32c6058a10b2d197c4f8b68cc1a3bde98ae5a26f67073a122e385ec8240b3: Status 404 returned error can't find the container with id 98c32c6058a10b2d197c4f8b68cc1a3bde98ae5a26f67073a122e385ec8240b3
Jul 01 14:38:44.234684 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.234489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl"
Jul 01 14:38:44.238471 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.238444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ch5j"
Jul 01 14:38:44.244101 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.244076    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode079a600_38ba_410b_949c_40bf6ef35e6f.slice/crio-af06995327d3e53ddeacce9f0a5fb2aa41eaa4335ffc4c99b0b968ed79629a5f WatchSource:0}: Error finding container af06995327d3e53ddeacce9f0a5fb2aa41eaa4335ffc4c99b0b968ed79629a5f: Status 404 returned error can't find the container with id af06995327d3e53ddeacce9f0a5fb2aa41eaa4335ffc4c99b0b968ed79629a5f
Jul 01 14:38:44.248326 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.248243    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26da1d3_b884_4104_9ec4_c0e1bcf3c333.slice/crio-03aeda703a012e35c8bfabafd8b59bce545deff23961ec1af29ac51221118a95 WatchSource:0}: Error finding container 03aeda703a012e35c8bfabafd8b59bce545deff23961ec1af29ac51221118a95: Status 404 returned error can't find the container with id 03aeda703a012e35c8bfabafd8b59bce545deff23961ec1af29ac51221118a95
Jul 01 14:38:44.252136 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:38:44.252110    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bbda80_5433_4a27_bfc3_a894363c3548.slice/crio-d55b5fe56d60bbd022ea6f84b113aead94c5cd84a442cf71dcd5e316b3b35857 WatchSource:0}: Error finding container d55b5fe56d60bbd022ea6f84b113aead94c5cd84a442cf71dcd5e316b3b35857: Status 404 returned error can't find the container with id d55b5fe56d60bbd022ea6f84b113aead94c5cd84a442cf71dcd5e316b3b35857
Jul 01 14:38:44.610759 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.610662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:44.610759 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.610726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:44.610759 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.610751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610851    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610875    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610875    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610885    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610942    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:38:45.61092055 +0000 UTC m=+4.137204554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610963    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:38:45.610954296 +0000 UTC m=+4.137238280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.610875    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:44.611053 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.611013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:38:45.610996092 +0000 UTC m=+4.137280088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:44.947597 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.947466    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 14:33:44 +0000 UTC" deadline="2028-03-17 20:24:16.126480644 +0000 UTC"
Jul 01 14:38:44.947597 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.947535    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15005h45m31.178950293s"
Jul 01 14:38:44.951784 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.951755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:44.951927 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:44.951899    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:44.963363 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.963140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bwfq" event={"ID":"e079a600-38ba-410b-949c-40bf6ef35e6f","Type":"ContainerStarted","Data":"af06995327d3e53ddeacce9f0a5fb2aa41eaa4335ffc4c99b0b968ed79629a5f"}
Jul 01 14:38:44.964801 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.964724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wdmbn" event={"ID":"c02426cb-5b55-4f73-ab60-16df7c4dcc2a","Type":"ContainerStarted","Data":"cdc313bed805eb8548f12e6df94cebde3991c10ad5c739a739d86597c8c1079c"}
Jul 01 14:38:44.966752 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.966712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hj4w7" event={"ID":"4cb2984d-8776-4605-8c15-d0cc4167c2ef","Type":"ContainerStarted","Data":"98c32c6058a10b2d197c4f8b68cc1a3bde98ae5a26f67073a122e385ec8240b3"}
Jul 01 14:38:44.969008 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.968982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swpjb" event={"ID":"6f9e4965-e925-4022-a963-8a03bc3b2c3c","Type":"ContainerStarted","Data":"eb6ac7909a200c91b11dd14fc040bdd223b8521b84071f2667d6fd049bdb3853"}
Jul 01 14:38:44.970378 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.970328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9p5nd" event={"ID":"35978eac-71cf-4911-a661-c713ab08588d","Type":"ContainerStarted","Data":"9922c222c8ab9acc43d83baff3ca72ef0323fea51deb6c35e3eb20dd6d30bc8d"}
Jul 01 14:38:44.971802 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.971779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ch5j" event={"ID":"c0bbda80-5433-4a27-bfc3-a894363c3548","Type":"ContainerStarted","Data":"d55b5fe56d60bbd022ea6f84b113aead94c5cd84a442cf71dcd5e316b3b35857"}
Jul 01 14:38:44.973239 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.973217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl" event={"ID":"a26da1d3-b884-4104-9ec4-c0e1bcf3c333","Type":"ContainerStarted","Data":"03aeda703a012e35c8bfabafd8b59bce545deff23961ec1af29ac51221118a95"}
Jul 01 14:38:44.975613 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.975555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerStarted","Data":"88c7a9d46e2759002e721ebf8ae6343733306a98547b2df08debe6f1edcf5381"}
Jul 01 14:38:44.978304 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:44.978222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"12fcb16028897e915db125a83a48e3855452c8de9465a319c5cc577d7c55b221"}
Jul 01 14:38:45.621054 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.621013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:45.621236 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.621088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:45.621236 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.621148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:45.621236 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621219    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621257    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621276    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621295    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621300    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:38:47.621282016 +0000 UTC m=+6.147565999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621310    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621349    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:38:47.621307266 +0000 UTC m=+6.147591268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:45.621450 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.621382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:38:47.621365664 +0000 UTC m=+6.147649665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:45.948538 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.948431    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 14:33:44 +0000 UTC" deadline="2028-02-29 15:52:28.822411879 +0000 UTC"
Jul 01 14:38:45.948538 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.948502    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14593h13m42.873914155s"
Jul 01 14:38:45.953023 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.952498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:45.953023 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.952638    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:45.954333 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:45.954309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:45.954456 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:45.954436    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:46.952338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:46.951699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:46.952338 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:46.951897    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:47.638834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:47.638792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:47.639048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:47.638857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:47.639048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:47.638899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:47.639173 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639049    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:38:47.639173 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639069    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:38:47.639173 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639082    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:47.639173 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:38:51.63912762 +0000 UTC m=+10.165411612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:47.639601 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639579    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:47.639697 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639636    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:38:51.63961825 +0000 UTC m=+10.165902239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:47.639780 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639697    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:47.639780 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.639732    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:38:51.639721091 +0000 UTC m=+10.166005080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:47.953185 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:47.952409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:47.953185 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.952614    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:47.953185 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:47.952978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:47.953185 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:47.953078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:48.952165 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:48.952027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:48.952342 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:48.952173    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:49.952054 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:49.951509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:49.952054 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:49.951675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:49.952054 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:49.951785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:49.952054 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:49.951669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:50.951774 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:50.951737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:50.951969 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:50.951873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:51.676989 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:51.676950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:51.677013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:51.677054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677121    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677168    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:38:59.67717725 +0000 UTC m=+18.203461252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677228    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:38:59.677211059 +0000 UTC m=+18.203495056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677273    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677294    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677320    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:51.677485 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.677367    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:38:59.677354031 +0000 UTC m=+18.203638027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:51.952400 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:51.952364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:51.952586 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.952515    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:51.952586 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:51.952559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:51.952692 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:51.952666    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:52.952333 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:52.952283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:52.952819 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:52.952443    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:53.953334 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:53.951689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:53.953334 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:53.951841    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:53.953334 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:53.952253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:53.953334 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:53.952348    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:54.951609 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:54.951575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:54.951773 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:54.951717    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:55.954859 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:55.954781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:55.955285 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:55.954948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:55.955285 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:55.954953    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:55.955285 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:55.955055    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:56.951756 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:56.951714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:56.951928 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:56.951836    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:57.951698 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:57.951663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:57.952170 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:57.951775    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:57.952170 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:57.951826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:57.952170 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:57.951947    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:38:58.952264 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:58.952225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:58.952772 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:58.952362    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:38:59.739724 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:59.739689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:59.739724 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:59.739732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:59.739755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739847    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739867    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739869    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:39:15.739900057 +0000 UTC m=+34.266184056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739940    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:15.739930368 +0000 UTC m=+34.266214356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739920    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:38:59.739962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.739958    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:59.740287 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.740001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:39:15.739990957 +0000 UTC m=+34.266274943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:38:59.951676 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:59.951647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:38:59.951825 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.951760    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:38:59.951899 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:38:59.951841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:38:59.951979 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:38:59.951957    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:00.951578 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:00.951541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:00.952028 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:00.951685    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:01.952118 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:01.952081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:01.952543 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:01.952190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:01.952543 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:01.952250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:01.952543 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:01.952372    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:02.952396 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:02.952352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:02.952830 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:02.952498    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:03.952328 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:03.952234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:03.952474 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:03.952234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:03.952474 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:03.952400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:03.952474 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:03.952429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:04.952493 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:04.952329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:04.953125 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:04.952566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:05.023744 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.023659    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e44db6a-a5cd-430e-9f12-fb6ce04488ba" containerID="1a95e606dde977d7705f3ee127e5d58637bb12f63d85c5db17ec57b3c96b4c02" exitCode=0
Jul 01 14:39:05.023892 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.023753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerDied","Data":"1a95e606dde977d7705f3ee127e5d58637bb12f63d85c5db17ec57b3c96b4c02"}
Jul 01 14:39:05.026999 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.026982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:39:05.027384 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.027362    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d360fed-4183-4b87-b1b1-63d8c7964670" containerID="24d1a062f99328322cdb23b1b8d5a2620c670157e2de6c64eb2d99786b25312a" exitCode=1
Jul 01 14:39:05.027462 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.027436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"0fed75b8ca88687e29e9300d7c7c49484bb36de49a75535317cca6586a80f367"}
Jul 01 14:39:05.027526 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.027513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"15706b50009cefd1c0df935856de37f7a2c3d29fba41bf5abf0f8ee04794124e"}
Jul 01 14:39:05.027580 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.027565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"f9916aa2d413aec477f2572b088c017cb7382961069bad15a2a0b0bf0ae67142"}
Jul 01 14:39:05.027647 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.027634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"a47dbdd73b68dd92be41db1799b63fccb3a3b5b15ff538c181ee3a2a89e881d8"}
Jul 01 14:39:05.030698 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.028275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerDied","Data":"24d1a062f99328322cdb23b1b8d5a2620c670157e2de6c64eb2d99786b25312a"}
Jul 01 14:39:05.030698 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.028381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"1f52a65090e094269fcdf13a0bdece34007f4ff023cfb2a59244c85cd080008d"}
Jul 01 14:39:05.032114 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.032091    2579 generic.go:356] "Generic (PLEG): container finished" podID="8758b1976ce38078ba9ae8f6eae9ad37" containerID="3caf0a006669886a3d0ca833df4b345a29aa001a100bf2f732d4a144619370b4" exitCode=0
Jul 01 14:39:05.032209 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.032178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" event={"ID":"8758b1976ce38078ba9ae8f6eae9ad37","Type":"ContainerDied","Data":"3caf0a006669886a3d0ca833df4b345a29aa001a100bf2f732d4a144619370b4"}
Jul 01 14:39:05.036004 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.035976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bwfq" event={"ID":"e079a600-38ba-410b-949c-40bf6ef35e6f","Type":"ContainerStarted","Data":"4eef6e3f04eb405953e0cfb9c76a568b54eef943f8d5f0adf5aca338d85f633a"}
Jul 01 14:39:05.037646 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.037626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wdmbn" event={"ID":"c02426cb-5b55-4f73-ab60-16df7c4dcc2a","Type":"ContainerStarted","Data":"37c0bfae56f4cc7c788eabcbd38c9977448fd5cf81701e5e26aeb6169f81bcae"}
Jul 01 14:39:05.038917 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.038894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swpjb" event={"ID":"6f9e4965-e925-4022-a963-8a03bc3b2c3c","Type":"ContainerStarted","Data":"05611dca62032379f6e06e4945e7dd7088465db30fb84798ba007f7d5e2dca9f"}
Jul 01 14:39:05.040195 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.040176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9p5nd" event={"ID":"35978eac-71cf-4911-a661-c713ab08588d","Type":"ContainerStarted","Data":"e6a11cdfec6c7325440c8e5b42991a20834045d10fbd13f00c0a1db1f5553639"}
Jul 01 14:39:05.041512 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.041493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" event={"ID":"371354312e904fda882e14f06ac6f309","Type":"ContainerStarted","Data":"a5c58bf0e212b416de2fb3b58f9443c268fb23f533802308a9a70e9379a4d497"}
Jul 01 14:39:05.043024 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.042998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ch5j" event={"ID":"c0bbda80-5433-4a27-bfc3-a894363c3548","Type":"ContainerStarted","Data":"cbaa524eeb51d71bc3f2008635dbcba2e6174f0a6a6c58c0f693ff74364168ac"}
Jul 01 14:39:05.044214 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.044196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl" event={"ID":"a26da1d3-b884-4104-9ec4-c0e1bcf3c333","Type":"ContainerStarted","Data":"cb7eb7789b6f40b7caf1332e4585fb3192e1fb6390d9653dacd59401ac2b7962"}
Jul 01 14:39:05.224024 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.223948    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9p5nd" podStartSLOduration=7.526460439 podStartE2EDuration="23.22392581s" totalImagesPullingTime="15.697465371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:05.189148113 +0000 UTC m=+23.715432119" watchObservedRunningTime="2026-07-01 14:39:05.22392581 +0000 UTC m=+23.750209815"
Jul 01 14:39:05.264453 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.264391    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wdmbn" podStartSLOduration=3.550464437 podStartE2EDuration="23.264368106s" totalImagesPullingTime="19.713903669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:05.224057007 +0000 UTC m=+23.750341012" watchObservedRunningTime="2026-07-01 14:39:05.264368106 +0000 UTC m=+23.790652113"
Jul 01 14:39:05.264664 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.264642    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8bwfq" podStartSLOduration=3.54384558 podStartE2EDuration="23.264636727s" totalImagesPullingTime="19.720791147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:05.264065733 +0000 UTC m=+23.790349740" watchObservedRunningTime="2026-07-01 14:39:05.264636727 +0000 UTC m=+23.790920730"
Jul 01 14:39:05.357978 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.357929    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" podStartSLOduration=22.357912757 podStartE2EDuration="22.357912757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:05.294459267 +0000 UTC m=+23.820743284" watchObservedRunningTime="2026-07-01 14:39:05.357912757 +0000 UTC m=+23.884196760"
Jul 01 14:39:05.409158 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.409107    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-swpjb" podStartSLOduration=3.237272357 podStartE2EDuration="23.409090724s" totalImagesPullingTime="20.171818367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:05.358964326 +0000 UTC m=+23.885248340" watchObservedRunningTime="2026-07-01 14:39:05.409090724 +0000 UTC m=+23.935374727"
Jul 01 14:39:05.439706 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.439658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ch5j" podStartSLOduration=3.724518645 podStartE2EDuration="23.439642759s" totalImagesPullingTime="19.715124114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:05.408864601 +0000 UTC m=+23.935148605" watchObservedRunningTime="2026-07-01 14:39:05.439642759 +0000 UTC m=+23.965926762"
Jul 01 14:39:05.951772 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.951714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:05.951772 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:05.951745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:05.951963 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:05.951846    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:05.951963 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:05.951933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:06.049293 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.049229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" event={"ID":"8758b1976ce38078ba9ae8f6eae9ad37","Type":"ContainerStarted","Data":"488f78639f540b63c0c58d53c5eed1190db50bcce7f8c15a3af693d97fe5faf4"}
Jul 01 14:39:06.051156 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.051103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hj4w7" event={"ID":"4cb2984d-8776-4605-8c15-d0cc4167c2ef","Type":"ContainerStarted","Data":"985976472b8599c7eab06f2f19523c327a3554706d4e0c18bdf80a09066e2aaa"}
Jul 01 14:39:06.106829 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.106796    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 14:39:06.124979 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.124867    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hj4w7" podStartSLOduration=4.703471809 podStartE2EDuration="24.124848318s" totalImagesPullingTime="19.421376509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:06.124593108 +0000 UTC m=+24.650877124" watchObservedRunningTime="2026-07-01 14:39:06.124848318 +0000 UTC m=+24.651132323"
Jul 01 14:39:06.125725 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.125695    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" podStartSLOduration=23.125684336 podStartE2EDuration="23.125684336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:06.090465872 +0000 UTC m=+24.616749877" watchObservedRunningTime="2026-07-01 14:39:06.125684336 +0000 UTC m=+24.651968341"
Jul 01 14:39:06.952475 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.952446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:06.952725 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:06.952564    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:06.965116 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.965016    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T14:39:06.106824365Z","UUID":"7a08dfdb-7b15-449d-992d-0504aecc62f7","Handler":null,"Name":"","Endpoint":""}
Jul 01 14:39:06.966897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.966878    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 14:39:06.967016 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:06.966921    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 14:39:07.055537 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:07.055461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl" event={"ID":"a26da1d3-b884-4104-9ec4-c0e1bcf3c333","Type":"ContainerStarted","Data":"5420e53a6f5dd2d4e37c9a2ccc4c887ea87fa2471c0485f0c8b7b127a3296487"}
Jul 01 14:39:07.058921 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:07.058901    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:39:07.059320 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:07.059274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"1e5d40bd927c422b73fdc0165905d351c77611c805405835068f895c28d9ec6d"}
Jul 01 14:39:07.952045 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:07.952012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:07.952264 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:07.952025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:07.952264 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:07.952153    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:07.952264 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:07.952183    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:08.064248 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:08.064203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl" event={"ID":"a26da1d3-b884-4104-9ec4-c0e1bcf3c333","Type":"ContainerStarted","Data":"71367ff1a72ff1457cdd758e2b16bd1240be59eef4a5effbe67d2e56ca75cd0c"}
Jul 01 14:39:08.951835 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:08.951805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:08.952016 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:08.951922    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:09.062995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.062959    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:39:09.063668 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.063636    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:39:09.066381 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.066353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:39:09.066804 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.066788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wdmbn"
Jul 01 14:39:09.087269 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.087212    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p7xxl" podStartSLOduration=4.85528124 podStartE2EDuration="27.08719336s" totalImagesPullingTime="22.23191212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:08.090594567 +0000 UTC m=+26.616878572" watchObservedRunningTime="2026-07-01 14:39:09.08719336 +0000 UTC m=+27.613477358"
Jul 01 14:39:09.954903 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.954871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:09.955001 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:09.954968    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:09.955040 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:09.955015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:09.955115 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:09.955097    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:10.070493 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:10.070417    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e44db6a-a5cd-430e-9f12-fb6ce04488ba" containerID="b376b724784d4c9b68b9ef09593f5bc1ce5facce56d1ca58d385c1eea94e1577" exitCode=0
Jul 01 14:39:10.071196 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:10.070507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerDied","Data":"b376b724784d4c9b68b9ef09593f5bc1ce5facce56d1ca58d385c1eea94e1577"}
Jul 01 14:39:10.074048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:10.074027    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:39:10.074431 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:10.074408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"07ce3a8f5d984325b38affa859b68df259fe4d6833277040b233aaef69018d21"}
Jul 01 14:39:10.074830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:10.074813    2579 scope.go:117] "RemoveContainer" containerID="24d1a062f99328322cdb23b1b8d5a2620c670157e2de6c64eb2d99786b25312a"
Jul 01 14:39:10.951613 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:10.951433    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:10.951899 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:10.951704    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:11.081338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.081298    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:39:11.081809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.081724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" event={"ID":"1d360fed-4183-4b87-b1b1-63d8c7964670","Type":"ContainerStarted","Data":"bd7284e37190f62a8b99b797a2e6e18116d587b13caa351d87d5457c735a2466"}
Jul 01 14:39:11.082150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.082131    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:39:11.082255 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.082229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:39:11.082330 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.082289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:39:11.100811 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.100768    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:39:11.106152 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.106126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:39:11.130503 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.130454    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb" podStartSLOduration=9.33312638 podStartE2EDuration="29.130438517s" totalImagesPullingTime="19.797312137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:11.129927542 +0000 UTC m=+29.656211546" watchObservedRunningTime="2026-07-01 14:39:11.130438517 +0000 UTC m=+29.656722537"
Jul 01 14:39:11.767016 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.766976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vwzh5"]
Jul 01 14:39:11.767197 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.767097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:11.767243 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:11.767190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:11.770534 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.770508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xl6ft"]
Jul 01 14:39:11.770651 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.770622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:11.770756 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:11.770735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:11.771151 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.771129    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljxh4"]
Jul 01 14:39:11.771238 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:11.771223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:11.771329 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:11.771293    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:12.085461 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:12.085427    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e44db6a-a5cd-430e-9f12-fb6ce04488ba" containerID="0c645381b8e703ffea73cba705e5282b778f704642fae3310444a220f50fc355" exitCode=0
Jul 01 14:39:12.085988 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:12.085507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerDied","Data":"0c645381b8e703ffea73cba705e5282b778f704642fae3310444a220f50fc355"}
Jul 01 14:39:12.951846 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:12.951813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:12.951999 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:12.951960    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:13.952653 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:13.952482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:13.953130 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:13.952559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:13.953130 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:13.952763    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:13.953130 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:13.952839    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:14.091910 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:14.091877    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e44db6a-a5cd-430e-9f12-fb6ce04488ba" containerID="2ba1ad7534b06f670561b90864469daabb329b6b23afd1be92f72f7c94b60024" exitCode=0
Jul 01 14:39:14.092076 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:14.091938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerDied","Data":"2ba1ad7534b06f670561b90864469daabb329b6b23afd1be92f72f7c94b60024"}
Jul 01 14:39:14.952030 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:14.951998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:14.952191 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:14.952131    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:15.754611 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:15.754579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:15.754634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:15.754672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754772    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754814    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754854    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:39:47.754831766 +0000 UTC m=+66.281115754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754819    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754873    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret podName:7c48eacc-cd11-4893-94ef-d8368f954658 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:47.754863071 +0000 UTC m=+66.281147059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret") pod "global-pull-secret-syncer-xl6ft" (UID: "7c48eacc-cd11-4893-94ef-d8368f954658") : object "kube-system"/"original-pull-secret" not registered
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754880    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754894    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mptwj for pod openshift-network-diagnostics/network-check-target-ljxh4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:15.755052 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.754944    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj podName:dfe1e7e8-3db4-4a7b-b138-27044da30c8d nodeName:}" failed. No retries permitted until 2026-07-01 14:39:47.754925935 +0000 UTC m=+66.281209934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mptwj" (UniqueName: "kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj") pod "network-check-target-ljxh4" (UID: "dfe1e7e8-3db4-4a7b-b138-27044da30c8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 14:39:15.952399 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:15.952357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:15.952399 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:15.952385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:15.952619 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.952517    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl6ft" podUID="7c48eacc-cd11-4893-94ef-d8368f954658"
Jul 01 14:39:15.952682 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:15.952655    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljxh4" podUID="dfe1e7e8-3db4-4a7b-b138-27044da30c8d"
Jul 01 14:39:16.952490 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:16.952459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:16.953074 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:16.952579    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vwzh5" podUID="12300cae-4145-449e-9bac-bf1b507ab3da"
Jul 01 14:39:17.326161 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.326130    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeReady"
Jul 01 14:39:17.326362 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.326327    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 14:39:17.378480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.378447    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g6n5t"]
Jul 01 14:39:17.381130 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.381105    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nfp67"]
Jul 01 14:39:17.381280 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.381225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.382830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.382809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.383693 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.383673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 14:39:17.383797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.383679    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4bfzd\""
Jul 01 14:39:17.383797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.383723    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 14:39:17.384797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.384768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kn75c\""
Jul 01 14:39:17.385271 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.385253    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 14:39:17.385471 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.385454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 14:39:17.385756 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.385739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 14:39:17.394847 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.394825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g6n5t"]
Jul 01 14:39:17.398404 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.398380    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nfp67"]
Jul 01 14:39:17.467921 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.467884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45fx\" (UniqueName: \"kubernetes.io/projected/6e3eb182-9ffa-4ca8-8b49-9c0683c23072-kube-api-access-b45fx\") pod \"ingress-canary-nfp67\" (UID: \"6e3eb182-9ffa-4ca8-8b49-9c0683c23072\") " pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.467921 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.467927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa0250-95e6-4f97-9688-6c72a2910812-tmp-dir\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.468169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.467963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fa0250-95e6-4f97-9688-6c72a2910812-config-volume\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.468169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.467983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.468169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.468005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr8r\" (UniqueName: \"kubernetes.io/projected/e4fa0250-95e6-4f97-9688-6c72a2910812-kube-api-access-dtr8r\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.468169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.468099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e3eb182-9ffa-4ca8-8b49-9c0683c23072-cert\") pod \"ingress-canary-nfp67\" (UID: \"6e3eb182-9ffa-4ca8-8b49-9c0683c23072\") " pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.568817 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.568778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45fx\" (UniqueName: \"kubernetes.io/projected/6e3eb182-9ffa-4ca8-8b49-9c0683c23072-kube-api-access-b45fx\") pod \"ingress-canary-nfp67\" (UID: \"6e3eb182-9ffa-4ca8-8b49-9c0683c23072\") " pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.568817 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.568820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa0250-95e6-4f97-9688-6c72a2910812-tmp-dir\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.569080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.568846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fa0250-95e6-4f97-9688-6c72a2910812-config-volume\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.569080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.569023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.569080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.569062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr8r\" (UniqueName: \"kubernetes.io/projected/e4fa0250-95e6-4f97-9688-6c72a2910812-kube-api-access-dtr8r\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.569232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.569111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e3eb182-9ffa-4ca8-8b49-9c0683c23072-cert\") pod \"ingress-canary-nfp67\" (UID: \"6e3eb182-9ffa-4ca8-8b49-9c0683c23072\") " pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.569232 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:17.569182    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:17.569232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.569202    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa0250-95e6-4f97-9688-6c72a2910812-tmp-dir\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.569363 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:17.569249    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:18.069227528 +0000 UTC m=+36.595511515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:17.569508 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.569486    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4fa0250-95e6-4f97-9688-6c72a2910812-config-volume\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.573707 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.573683    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e3eb182-9ffa-4ca8-8b49-9c0683c23072-cert\") pod \"ingress-canary-nfp67\" (UID: \"6e3eb182-9ffa-4ca8-8b49-9c0683c23072\") " pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.578442 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.578362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr8r\" (UniqueName: \"kubernetes.io/projected/e4fa0250-95e6-4f97-9688-6c72a2910812-kube-api-access-dtr8r\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:17.578671 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.578647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45fx\" (UniqueName: \"kubernetes.io/projected/6e3eb182-9ffa-4ca8-8b49-9c0683c23072-kube-api-access-b45fx\") pod \"ingress-canary-nfp67\" (UID: \"6e3eb182-9ffa-4ca8-8b49-9c0683c23072\") " pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.700923 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.700883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nfp67"
Jul 01 14:39:17.951862 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.951831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:17.952044 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.952021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:17.954480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.954455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 14:39:17.955013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.954531    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr22r\""
Jul 01 14:39:17.955013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.954753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 14:39:17.955013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:17.954826    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 14:39:18.073572 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:18.073536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:18.073738 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:18.073666    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:18.073738 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:18.073719    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:19.073706884 +0000 UTC m=+37.599990866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:18.331108 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:18.331069    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nfp67"]
Jul 01 14:39:18.334856 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:18.334831    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3eb182_9ffa_4ca8_8b49_9c0683c23072.slice/crio-a6adbe19a2196e74dd0eb03eedac08d1aca7fef836094242cd3531db7676e133 WatchSource:0}: Error finding container a6adbe19a2196e74dd0eb03eedac08d1aca7fef836094242cd3531db7676e133: Status 404 returned error can't find the container with id a6adbe19a2196e74dd0eb03eedac08d1aca7fef836094242cd3531db7676e133
Jul 01 14:39:18.952481 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:18.952443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:18.954966 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:18.954939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 14:39:18.955438 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:18.954975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5f8j\""
Jul 01 14:39:19.080593 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:19.080563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:19.080789 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:19.080742    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:19.080851 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:19.080823    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:21.080800653 +0000 UTC m=+39.607084637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:19.104634 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:19.104599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nfp67" event={"ID":"6e3eb182-9ffa-4ca8-8b49-9c0683c23072","Type":"ContainerStarted","Data":"a6adbe19a2196e74dd0eb03eedac08d1aca7fef836094242cd3531db7676e133"}
Jul 01 14:39:19.107619 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:19.107581    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e44db6a-a5cd-430e-9f12-fb6ce04488ba" containerID="8e4423fd355ce38e0177c7983a2d95a6e3c0a280dde2c8be69baabfc97a1bc94" exitCode=0
Jul 01 14:39:19.107747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:19.107628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerDied","Data":"8e4423fd355ce38e0177c7983a2d95a6e3c0a280dde2c8be69baabfc97a1bc94"}
Jul 01 14:39:20.113268 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:20.113234    2579 generic.go:356] "Generic (PLEG): container finished" podID="3e44db6a-a5cd-430e-9f12-fb6ce04488ba" containerID="605b1fdbb9ede2dd4c95e0739ad05dbb3c7addd482a797e37eb8b156df1559c9" exitCode=0
Jul 01 14:39:20.113780 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:20.113336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerDied","Data":"605b1fdbb9ede2dd4c95e0739ad05dbb3c7addd482a797e37eb8b156df1559c9"}
Jul 01 14:39:21.095851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:21.095697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:21.095984 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:21.095878    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:21.095984 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:21.095959    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:25.095937492 +0000 UTC m=+43.622221487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:21.117624 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:21.117594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nfp67" event={"ID":"6e3eb182-9ffa-4ca8-8b49-9c0683c23072","Type":"ContainerStarted","Data":"f4ed4e955926f810a3a96e96ffaf8a9d19e32f3a8877e13e4f7c57761b3297f1"}
Jul 01 14:39:21.121003 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:21.120981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" event={"ID":"3e44db6a-a5cd-430e-9f12-fb6ce04488ba","Type":"ContainerStarted","Data":"adb3a001e31faca83eeeaece1651144a1b828db221c39d0eca31a9ea2bd33df5"}
Jul 01 14:39:21.148227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:21.148164    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bz8l6" podStartSLOduration=8.278563954 podStartE2EDuration="39.148143576s" totalImagesPullingTime="30.869579622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:21.146347256 +0000 UTC m=+39.672631261" watchObservedRunningTime="2026-07-01 14:39:21.148143576 +0000 UTC m=+39.674427578"
Jul 01 14:39:25.123687 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:25.123643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:25.124095 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:25.123796    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:25.124095 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:25.123861    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:33.123846309 +0000 UTC m=+51.650130315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:27.354579 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.354512    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nfp67" podStartSLOduration=7.698272241 podStartE2EDuration="10.35448941s" totalImagesPullingTime="2.656217169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:22.141682251 +0000 UTC m=+40.667966256" watchObservedRunningTime="2026-07-01 14:39:27.35448941 +0000 UTC m=+45.880773416"
Jul 01 14:39:27.355344 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.355280    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"]
Jul 01 14:39:27.391736 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.391707    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-l9tjh"]
Jul 01 14:39:27.391894 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.391837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.398840 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.398801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 01 14:39:27.398979 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.398961    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 14:39:27.399437 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.399419    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vtgkn\""
Jul 01 14:39:27.400904 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.400886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 14:39:27.400997 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.400916    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 01 14:39:27.417366 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.417340    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-674f7cb5-nhjq7"]
Jul 01 14:39:27.417530 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.417402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.423972 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.423949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 14:39:27.424072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.423950    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 01 14:39:27.424241 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.424226    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 14:39:27.424392 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.424379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 01 14:39:27.424712 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.424698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tbjwf\""
Jul 01 14:39:27.432555 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.432538    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 01 14:39:27.435360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.435343    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"]
Jul 01 14:39:27.435458 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.435372    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-l9tjh"]
Jul 01 14:39:27.435458 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.435385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674f7cb5-nhjq7"]
Jul 01 14:39:27.435528 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.435464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.439426 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.439409    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 01 14:39:27.439649 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.439631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 01 14:39:27.439754 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.439736    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 01 14:39:27.439817 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.439779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 01 14:39:27.439872 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.439836    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 01 14:39:27.440353 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.440334    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 01 14:39:27.440767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.440749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5mkmz\""
Jul 01 14:39:27.542641 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-tmp\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.542641 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-snapshots\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.542832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvt22\" (UniqueName: \"kubernetes.io/projected/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-kube-api-access-pvt22\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.542832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.542832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-default-certificate\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.542832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrj8d\" (UniqueName: \"kubernetes.io/projected/e2242c5d-0e33-4904-bd89-024c85303ff3-kube-api-access-zrj8d\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.542960 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.542960 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.542960 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-serving-cert\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.542960 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2242c5d-0e33-4904-bd89-024c85303ff3-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.543078 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.542981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-stats-auth\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.543078 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.543013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.543078 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.543030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-service-ca-bundle\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.543078 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.543054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjn55\" (UniqueName: \"kubernetes.io/projected/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-kube-api-access-xjn55\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.643798 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.643765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.643950 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.643804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-serving-cert\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.643950 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.643832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2242c5d-0e33-4904-bd89-024c85303ff3-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.643950 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.643874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-stats-auth\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.643950 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:27.643930    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:27.644132 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:27.644014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls podName:e2242c5d-0e33-4904-bd89-024c85303ff3 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:28.14399201 +0000 UTC m=+46.670276010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-s4fnf" (UID: "e2242c5d-0e33-4904-bd89-024c85303ff3") : secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:27.644132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.644132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-service-ca-bundle\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.644132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjn55\" (UniqueName: \"kubernetes.io/projected/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-kube-api-access-xjn55\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.644360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-tmp\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.644360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-snapshots\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.644360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvt22\" (UniqueName: \"kubernetes.io/projected/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-kube-api-access-pvt22\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.644360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.644360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-default-certificate\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.644592 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrj8d\" (UniqueName: \"kubernetes.io/projected/e2242c5d-0e33-4904-bd89-024c85303ff3-kube-api-access-zrj8d\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.644592 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.644592 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:27.644471    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 14:39:27.644592 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:27.644533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:28.144516401 +0000 UTC m=+46.670800407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : secret "router-metrics-certs-default" not found
Jul 01 14:39:27.644592 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:27.644562    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:28.144545749 +0000 UTC m=+46.670829748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : configmap references non-existent config key: service-ca.crt
Jul 01 14:39:27.644838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-service-ca-bundle\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.644838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2242c5d-0e33-4904-bd89-024c85303ff3-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.644838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-snapshots\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.645002 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.644940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-tmp\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.645053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.645042    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.647861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.647829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-serving-cert\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.647967 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.647940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-stats-auth\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.647967 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.647947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-default-certificate\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.653813 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.653783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjn55\" (UniqueName: \"kubernetes.io/projected/c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255-kube-api-access-xjn55\") pod \"insights-operator-5f67b55555-l9tjh\" (UID: \"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255\") " pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.654042 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.654025    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvt22\" (UniqueName: \"kubernetes.io/projected/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-kube-api-access-pvt22\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:27.654719 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.654702    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrj8d\" (UniqueName: \"kubernetes.io/projected/e2242c5d-0e33-4904-bd89-024c85303ff3-kube-api-access-zrj8d\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:27.725966 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.725931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-l9tjh"
Jul 01 14:39:27.878759 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:27.878728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-l9tjh"]
Jul 01 14:39:27.882041 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:27.882014    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e4ba3b_10c7_4a6a_9d75_de2d5bc04255.slice/crio-a67da0762926a9562e0a3c87f18f82adcc562f9316d41d3a2119f77ee48e4922 WatchSource:0}: Error finding container a67da0762926a9562e0a3c87f18f82adcc562f9316d41d3a2119f77ee48e4922: Status 404 returned error can't find the container with id a67da0762926a9562e0a3c87f18f82adcc562f9316d41d3a2119f77ee48e4922
Jul 01 14:39:28.140029 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:28.139997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-l9tjh" event={"ID":"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255","Type":"ContainerStarted","Data":"a67da0762926a9562e0a3c87f18f82adcc562f9316d41d3a2119f77ee48e4922"}
Jul 01 14:39:28.148744 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:28.148721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:28.148826 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:28.148769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:28.148826 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:28.148804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:28.148905 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:28.148871    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 14:39:28.148905 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:28.148897    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:28.148962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:28.148935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:29.14891939 +0000 UTC m=+47.675203378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : secret "router-metrics-certs-default" not found
Jul 01 14:39:28.148962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:28.148950    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:29.148944216 +0000 UTC m=+47.675228198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : configmap references non-existent config key: service-ca.crt
Jul 01 14:39:28.148962 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:28.148962    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls podName:e2242c5d-0e33-4904-bd89-024c85303ff3 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:29.148956088 +0000 UTC m=+47.675240070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-s4fnf" (UID: "e2242c5d-0e33-4904-bd89-024c85303ff3") : secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:29.156426 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:29.156393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:29.156442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:29.156498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:29.156571    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:31.156553858 +0000 UTC m=+49.682837840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : configmap references non-existent config key: service-ca.crt
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:29.156604    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:29.156656    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls podName:e2242c5d-0e33-4904-bd89-024c85303ff3 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:31.156644655 +0000 UTC m=+49.682928641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-s4fnf" (UID: "e2242c5d-0e33-4904-bd89-024c85303ff3") : secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:29.156604    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 14:39:29.156792 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:29.156693    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:31.156683231 +0000 UTC m=+49.682967217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : secret "router-metrics-certs-default" not found
Jul 01 14:39:31.148681 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.148611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-l9tjh" event={"ID":"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255","Type":"ContainerStarted","Data":"29801719e19bf5610362edda0111909be01d586d612a64bb5fafe9d46d5b85fc"}
Jul 01 14:39:31.168975 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.168847    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-l9tjh" podStartSLOduration=1.157738443 podStartE2EDuration="4.168830846s" totalImagesPullingTime="3.011092403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:31.168485354 +0000 UTC m=+49.694769386" watchObservedRunningTime="2026-07-01 14:39:31.168830846 +0000 UTC m=+49.695114849"
Jul 01 14:39:31.172781 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.172753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:31.172876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.172815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:31.172916 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.172896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:31.172916 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.172909    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:35.172890641 +0000 UTC m=+53.699174632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : configmap references non-existent config key: service-ca.crt
Jul 01 14:39:31.172982 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.172963    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:31.172982 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.172977    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 14:39:31.173042 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.173020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls podName:e2242c5d-0e33-4904-bd89-024c85303ff3 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:35.17300478 +0000 UTC m=+53.699288764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-s4fnf" (UID: "e2242c5d-0e33-4904-bd89-024c85303ff3") : secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:31.173042 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.173037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:35.173029114 +0000 UTC m=+53.699313096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : secret "router-metrics-certs-default" not found
Jul 01 14:39:31.303059 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.303022    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"]
Jul 01 14:39:31.321498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.321464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"]
Jul 01 14:39:31.321663 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.321582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"
Jul 01 14:39:31.324082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.324057    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:39:31.324205 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.324093    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 01 14:39:31.324205 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.324117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9mgts\""
Jul 01 14:39:31.402616 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.402543    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"]
Jul 01 14:39:31.425179 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.425153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"]
Jul 01 14:39:31.425342 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.425251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:31.428364 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.428342    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 01 14:39:31.428516 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.428496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 01 14:39:31.428989 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.428931    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c2qqz\""
Jul 01 14:39:31.431741 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.429195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:39:31.475145 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.475103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnb6\" (UniqueName: \"kubernetes.io/projected/5595e14a-62bd-47b1-b12e-125d7ecbd24d-kube-api-access-ddnb6\") pod \"volume-data-source-validator-69d9f679db-p4zfr\" (UID: \"5595e14a-62bd-47b1-b12e-125d7ecbd24d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"
Jul 01 14:39:31.575577 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.575539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdxm\" (UniqueName: \"kubernetes.io/projected/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-kube-api-access-hsdxm\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:31.575746 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.575724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnb6\" (UniqueName: \"kubernetes.io/projected/5595e14a-62bd-47b1-b12e-125d7ecbd24d-kube-api-access-ddnb6\") pod \"volume-data-source-validator-69d9f679db-p4zfr\" (UID: \"5595e14a-62bd-47b1-b12e-125d7ecbd24d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"
Jul 01 14:39:31.575789 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.575758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:31.585317 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.585275    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnb6\" (UniqueName: \"kubernetes.io/projected/5595e14a-62bd-47b1-b12e-125d7ecbd24d-kube-api-access-ddnb6\") pod \"volume-data-source-validator-69d9f679db-p4zfr\" (UID: \"5595e14a-62bd-47b1-b12e-125d7ecbd24d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"
Jul 01 14:39:31.630255 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.630221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"
Jul 01 14:39:31.676785 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.676696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdxm\" (UniqueName: \"kubernetes.io/projected/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-kube-api-access-hsdxm\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:31.676941 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.676877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:31.677016 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.677000    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 14:39:31.677084 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:31.677072    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls podName:c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac nodeName:}" failed. No retries permitted until 2026-07-01 14:39:32.177051859 +0000 UTC m=+50.703335861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cng95" (UID: "c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac") : secret "samples-operator-tls" not found
Jul 01 14:39:31.686117 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.686090    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdxm\" (UniqueName: \"kubernetes.io/projected/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-kube-api-access-hsdxm\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:31.768129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:31.768095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr"]
Jul 01 14:39:31.771167 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:31.771129    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5595e14a_62bd_47b1_b12e_125d7ecbd24d.slice/crio-aa714662e4c0668bedf364003afe98a7e794d69606ee4c4ec6633c1a2f96b868 WatchSource:0}: Error finding container aa714662e4c0668bedf364003afe98a7e794d69606ee4c4ec6633c1a2f96b868: Status 404 returned error can't find the container with id aa714662e4c0668bedf364003afe98a7e794d69606ee4c4ec6633c1a2f96b868
Jul 01 14:39:32.152220 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.152178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr" event={"ID":"5595e14a-62bd-47b1-b12e-125d7ecbd24d","Type":"ContainerStarted","Data":"aa714662e4c0668bedf364003afe98a7e794d69606ee4c4ec6633c1a2f96b868"}
Jul 01 14:39:32.180982 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.180949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:32.181137 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:32.181074    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 14:39:32.181189 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:32.181142    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls podName:c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac nodeName:}" failed. No retries permitted until 2026-07-01 14:39:33.181122129 +0000 UTC m=+51.707406126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cng95" (UID: "c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac") : secret "samples-operator-tls" not found
Jul 01 14:39:32.769448 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.768336    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mpkp9"]
Jul 01 14:39:32.778207 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.778184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.781097 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.781072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g767b\""
Jul 01 14:39:32.781227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.781127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpkp9"]
Jul 01 14:39:32.781321 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.781281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 14:39:32.781505 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.781486    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 14:39:32.784594 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.784568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3bec9de-cb72-4480-a078-60a9ab73b889-data-volume\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.784700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.784608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g86v\" (UniqueName: \"kubernetes.io/projected/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-api-access-6g86v\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.784700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.784639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.784700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.784696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3bec9de-cb72-4480-a078-60a9ab73b889-crio-socket\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.785006 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.784755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.885805 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.885764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3bec9de-cb72-4480-a078-60a9ab73b889-data-volume\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.885805 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.885811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g86v\" (UniqueName: \"kubernetes.io/projected/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-api-access-6g86v\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.886045 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.885837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.886045 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.885891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3bec9de-cb72-4480-a078-60a9ab73b889-crio-socket\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.886045 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.885939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.886045 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:32.886015    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:32.886226 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:32.886084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:33.386063205 +0000 UTC m=+51.912347208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:32.886226 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:32.886098    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 14:39:32.886226 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:32.886159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:33.386145491 +0000 UTC m=+51.912429492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : configmap "kube-rbac-proxy" not found
Jul 01 14:39:32.886226 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.886173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3bec9de-cb72-4480-a078-60a9ab73b889-data-volume\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.886440 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.886280    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3bec9de-cb72-4480-a078-60a9ab73b889-crio-socket\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:32.896101 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:32.896077    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g86v\" (UniqueName: \"kubernetes.io/projected/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-api-access-6g86v\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:33.187980 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:33.187940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:33.188353 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.188102    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:33.188353 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:33.188155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:33.188353 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.188172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:49.188150916 +0000 UTC m=+67.714434920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:33.188353 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.188236    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 14:39:33.188353 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.188282    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls podName:c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac nodeName:}" failed. No retries permitted until 2026-07-01 14:39:35.188269037 +0000 UTC m=+53.714553025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cng95" (UID: "c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac") : secret "samples-operator-tls" not found
Jul 01 14:39:33.389467 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:33.389434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:33.389635 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:33.389494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:33.389635 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.389580    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 14:39:33.389635 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.389620    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:33.389768 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.389640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:34.38962411 +0000 UTC m=+52.915908096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : configmap "kube-rbac-proxy" not found
Jul 01 14:39:33.389768 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:33.389685    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:34.389667383 +0000 UTC m=+52.915951373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:34.159099 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:34.159063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr" event={"ID":"5595e14a-62bd-47b1-b12e-125d7ecbd24d","Type":"ContainerStarted","Data":"74308eb22cbec215e04fa7c213590ca440841949d91ab5977fd498a2d07c1410"}
Jul 01 14:39:34.182704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:34.182639    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-p4zfr" podStartSLOduration=1.6486020049999999 podStartE2EDuration="3.182622215s" totalImagesPullingTime="1.53402021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:34.181488663 +0000 UTC m=+52.707772679" watchObservedRunningTime="2026-07-01 14:39:34.182622215 +0000 UTC m=+52.708906197"
Jul 01 14:39:34.398080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:34.398045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:34.398466 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:34.398108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:34.398466 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:34.398210    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:34.398466 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:34.398236    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 14:39:34.398466 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:34.398292    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:36.39827221 +0000 UTC m=+54.924556206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : configmap "kube-rbac-proxy" not found
Jul 01 14:39:34.398466 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:34.398329    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:36.398319537 +0000 UTC m=+54.924603520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:34.642103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:34.642075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9p5nd_35978eac-71cf-4911-a661-c713ab08588d/dns-node-resolver/0.log"
Jul 01 14:39:35.043624 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:35.043556    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bwfq_e079a600-38ba-410b-949c-40bf6ef35e6f/node-ca/0.log"
Jul 01 14:39:35.204777 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:35.204742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:35.205025 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:35.204824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:35.205025 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:35.204854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:35.205025 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.204888    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 14:39:35.205025 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.204957    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:43.204942411 +0000 UTC m=+61.731226394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : secret "router-metrics-certs-default" not found
Jul 01 14:39:35.205025 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.204980    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:43.204967063 +0000 UTC m=+61.731251045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : configmap references non-existent config key: service-ca.crt
Jul 01 14:39:35.205025 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:35.204893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:35.205292 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.204985    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:35.205292 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.205022    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 14:39:35.205292 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.205159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls podName:e2242c5d-0e33-4904-bd89-024c85303ff3 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:43.205136752 +0000 UTC m=+61.731420949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-s4fnf" (UID: "e2242c5d-0e33-4904-bd89-024c85303ff3") : secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:35.205292 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:35.205178    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls podName:c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac nodeName:}" failed. No retries permitted until 2026-07-01 14:39:39.205170065 +0000 UTC m=+57.731454046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cng95" (UID: "c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac") : secret "samples-operator-tls" not found
Jul 01 14:39:36.043106 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.043067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nfp67_6e3eb182-9ffa-4ca8-8b49-9c0683c23072/serve-healthcheck-canary/0.log"
Jul 01 14:39:36.311918 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.311841    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"]
Jul 01 14:39:36.336569 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.336542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"]
Jul 01 14:39:36.336729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.336633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.345655 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.345629    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 01 14:39:36.345655 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.345650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 01 14:39:36.345799 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.345629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 01 14:39:36.346018 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.346003    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wc7fl\""
Jul 01 14:39:36.346062 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.346043    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 01 14:39:36.417222 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.417186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:36.417222 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.417230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsf6\" (UniqueName: \"kubernetes.io/projected/d359dafa-4332-46a3-ae6c-4d6197a474f6-kube-api-access-jzsf6\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.417516 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:36.417345    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 14:39:36.417516 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:36.417418    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:40.417397382 +0000 UTC m=+58.943681368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : configmap "kube-rbac-proxy" not found
Jul 01 14:39:36.417516 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.417494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:36.417678 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.417533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d359dafa-4332-46a3-ae6c-4d6197a474f6-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.417678 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.417590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d359dafa-4332-46a3-ae6c-4d6197a474f6-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.417678 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:36.417619    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:36.417763 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:36.417687    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:40.417671232 +0000 UTC m=+58.943955235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:36.518680 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.518644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsf6\" (UniqueName: \"kubernetes.io/projected/d359dafa-4332-46a3-ae6c-4d6197a474f6-kube-api-access-jzsf6\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.518845 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.518832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d359dafa-4332-46a3-ae6c-4d6197a474f6-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.518904 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.518868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d359dafa-4332-46a3-ae6c-4d6197a474f6-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.519515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.519498    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d359dafa-4332-46a3-ae6c-4d6197a474f6-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.521162 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.521147    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d359dafa-4332-46a3-ae6c-4d6197a474f6-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.531677 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.531648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsf6\" (UniqueName: \"kubernetes.io/projected/d359dafa-4332-46a3-ae6c-4d6197a474f6-kube-api-access-jzsf6\") pod \"kube-storage-version-migrator-operator-79964ff6f9-8dfkr\" (UID: \"d359dafa-4332-46a3-ae6c-4d6197a474f6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.645150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.645116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"
Jul 01 14:39:36.782898 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:36.782864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr"]
Jul 01 14:39:36.786037 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:36.786012    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd359dafa_4332_46a3_ae6c_4d6197a474f6.slice/crio-9aa9c760e4802ab7ee9e976a11f72ce87a46393a91939ad339fd87bc3f30548e WatchSource:0}: Error finding container 9aa9c760e4802ab7ee9e976a11f72ce87a46393a91939ad339fd87bc3f30548e: Status 404 returned error can't find the container with id 9aa9c760e4802ab7ee9e976a11f72ce87a46393a91939ad339fd87bc3f30548e
Jul 01 14:39:37.167102 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.167065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr" event={"ID":"d359dafa-4332-46a3-ae6c-4d6197a474f6","Type":"ContainerStarted","Data":"9aa9c760e4802ab7ee9e976a11f72ce87a46393a91939ad339fd87bc3f30548e"}
Jul 01 14:39:37.764143 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.764110    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57db6cfdbc-sp9r4"]
Jul 01 14:39:37.766381 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.766360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.768739 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.768716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9s77\""
Jul 01 14:39:37.768739 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.768730    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 14:39:37.768899 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.768756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 14:39:37.770120 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.770098    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 14:39:37.774703 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.774680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 14:39:37.782679 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.782652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57db6cfdbc-sp9r4"]
Jul 01 14:39:37.834874 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.834839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqp4\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-kube-api-access-qqqp4\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835041 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.834900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-ca-trust-extracted\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835041 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.835018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-bound-sa-token\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.835067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-trusted-ca\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.835106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-image-registry-private-configuration\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835201 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.835138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835251 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.835235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-certificates\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.835290 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.835259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-installation-pull-secrets\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936386 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-ca-trust-extracted\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936553 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-bound-sa-token\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936553 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-trusted-ca\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936553 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-image-registry-private-configuration\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936553 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-certificates\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-installation-pull-secrets\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqp4\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-kube-api-access-qqqp4\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.936747 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:37.936734    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 14:39:37.936946 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:37.936756    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57db6cfdbc-sp9r4: secret "image-registry-tls" not found
Jul 01 14:39:37.936946 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:37.936826    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls podName:f3be1174-0f05-4c7f-a7ca-4669b3b5467e nodeName:}" failed. No retries permitted until 2026-07-01 14:39:38.436804626 +0000 UTC m=+56.963088621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls") pod "image-registry-57db6cfdbc-sp9r4" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e") : secret "image-registry-tls" not found
Jul 01 14:39:37.936946 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.936823    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-ca-trust-extracted\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.937198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.937179    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-certificates\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.937407 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.937389    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-trusted-ca\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.938891 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.938865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-image-registry-private-configuration\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.939028 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.939013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-installation-pull-secrets\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.946986 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.946963    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqp4\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-kube-api-access-qqqp4\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:37.947892 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:37.947870    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-bound-sa-token\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:38.442169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:38.442133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:38.442647 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:38.442277    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 14:39:38.442647 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:38.442316    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57db6cfdbc-sp9r4: secret "image-registry-tls" not found
Jul 01 14:39:38.442647 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:38.442385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls podName:f3be1174-0f05-4c7f-a7ca-4669b3b5467e nodeName:}" failed. No retries permitted until 2026-07-01 14:39:39.442368782 +0000 UTC m=+57.968652763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls") pod "image-registry-57db6cfdbc-sp9r4" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e") : secret "image-registry-tls" not found
Jul 01 14:39:39.174178 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:39.174140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr" event={"ID":"d359dafa-4332-46a3-ae6c-4d6197a474f6","Type":"ContainerStarted","Data":"49f3c09ad11a8725716c3bf082e9a54e4dd6852a3cf98170a1b5826955fb1984"}
Jul 01 14:39:39.194249 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:39.194204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr" podStartSLOduration=1.010118831 podStartE2EDuration="3.194187792s" totalImagesPullingTime="2.184068961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:39.193385564 +0000 UTC m=+57.719669569" watchObservedRunningTime="2026-07-01 14:39:39.194187792 +0000 UTC m=+57.720471795"
Jul 01 14:39:39.251113 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:39.251076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:39.251264 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:39.251204    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 14:39:39.251327 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:39.251274    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls podName:c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac nodeName:}" failed. No retries permitted until 2026-07-01 14:39:47.251255405 +0000 UTC m=+65.777539410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cng95" (UID: "c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac") : secret "samples-operator-tls" not found
Jul 01 14:39:39.452371 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:39.452327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:39.452746 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:39.452451    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 14:39:39.452746 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:39.452468    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57db6cfdbc-sp9r4: secret "image-registry-tls" not found
Jul 01 14:39:39.452746 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:39.452561    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls podName:f3be1174-0f05-4c7f-a7ca-4669b3b5467e nodeName:}" failed. No retries permitted until 2026-07-01 14:39:41.452540716 +0000 UTC m=+59.978824700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls") pod "image-registry-57db6cfdbc-sp9r4" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e") : secret "image-registry-tls" not found
Jul 01 14:39:40.459820 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:40.459770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:40.460230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:40.459878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:40.460230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:40.459931    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:40.460230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:40.459996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:48.459983251 +0000 UTC m=+66.986267236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:40.460230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:40.460006    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 14:39:40.460230 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:40.460066    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:48.460048711 +0000 UTC m=+66.986332707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : configmap "kube-rbac-proxy" not found
Jul 01 14:39:41.469706 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:41.469652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:41.470120 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:41.469801    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 14:39:41.470120 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:41.469821    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57db6cfdbc-sp9r4: secret "image-registry-tls" not found
Jul 01 14:39:41.470120 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:41.469878    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls podName:f3be1174-0f05-4c7f-a7ca-4669b3b5467e nodeName:}" failed. No retries permitted until 2026-07-01 14:39:45.46986272 +0000 UTC m=+63.996146702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls") pod "image-registry-57db6cfdbc-sp9r4" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e") : secret "image-registry-tls" not found
Jul 01 14:39:43.102587 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:43.102562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpdnb"
Jul 01 14:39:43.284380 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:43.284344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:43.284585 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:43.284395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:43.284585 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:43.284474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:43.284585 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:43.284563    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 14:39:43.284585 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:43.284566    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:59.284544678 +0000 UTC m=+77.810828674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : configmap references non-existent config key: service-ca.crt
Jul 01 14:39:43.284585 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:43.284566    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:43.284790 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:43.284612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs podName:b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb nodeName:}" failed. No retries permitted until 2026-07-01 14:39:59.284600247 +0000 UTC m=+77.810884243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs") pod "router-default-674f7cb5-nhjq7" (UID: "b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb") : secret "router-metrics-certs-default" not found
Jul 01 14:39:43.284790 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:43.284624    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls podName:e2242c5d-0e33-4904-bd89-024c85303ff3 nodeName:}" failed. No retries permitted until 2026-07-01 14:39:59.284618576 +0000 UTC m=+77.810902559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-s4fnf" (UID: "e2242c5d-0e33-4904-bd89-024c85303ff3") : secret "cluster-monitoring-operator-tls" not found
Jul 01 14:39:45.506652 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:45.506615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:45.507043 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:45.506733    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 14:39:45.507043 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:45.506746    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57db6cfdbc-sp9r4: secret "image-registry-tls" not found
Jul 01 14:39:45.507043 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:45.506810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls podName:f3be1174-0f05-4c7f-a7ca-4669b3b5467e nodeName:}" failed. No retries permitted until 2026-07-01 14:39:53.506794246 +0000 UTC m=+72.033078248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls") pod "image-registry-57db6cfdbc-sp9r4" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e") : secret "image-registry-tls" not found
Jul 01 14:39:47.320749 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.320699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:39:47.321187 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:47.320847    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 14:39:47.321187 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:47.320929    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls podName:c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac nodeName:}" failed. No retries permitted until 2026-07-01 14:40:03.320914329 +0000 UTC m=+81.847198315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-cng95" (UID: "c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac") : secret "samples-operator-tls" not found
Jul 01 14:39:47.826031 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.825984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:39:47.826194 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.826054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:47.826194 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.826096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:47.828612 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.828590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 14:39:47.828699 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.828668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 14:39:47.828699 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.828680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 14:39:47.836663 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:47.836643    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 14:39:47.836723 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:47.836711    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs podName:12300cae-4145-449e-9bac-bf1b507ab3da nodeName:}" failed. No retries permitted until 2026-07-01 14:40:51.836694287 +0000 UTC m=+130.362978285 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs") pod "network-metrics-daemon-vwzh5" (UID: "12300cae-4145-449e-9bac-bf1b507ab3da") : secret "metrics-daemon-secret" not found
Jul 01 14:39:47.838729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.838705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 14:39:47.838842 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.838769    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c48eacc-cd11-4893-94ef-d8368f954658-original-pull-secret\") pod \"global-pull-secret-syncer-xl6ft\" (UID: \"7c48eacc-cd11-4893-94ef-d8368f954658\") " pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:47.848890 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.848855    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptwj\" (UniqueName: \"kubernetes.io/projected/dfe1e7e8-3db4-4a7b-b138-27044da30c8d-kube-api-access-mptwj\") pod \"network-check-target-ljxh4\" (UID: \"dfe1e7e8-3db4-4a7b-b138-27044da30c8d\") " pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:47.963417 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.963391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl6ft"
Jul 01 14:39:47.971699 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.971675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr22r\""
Jul 01 14:39:47.979735 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:47.979716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:48.096165 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.096095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xl6ft"]
Jul 01 14:39:48.131319 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.131278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljxh4"]
Jul 01 14:39:48.133853 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:48.133830    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe1e7e8_3db4_4a7b_b138_27044da30c8d.slice/crio-312b52ca954ecd83c4757005dd6c03e7333bba0b581cf266e720af088bd2247e WatchSource:0}: Error finding container 312b52ca954ecd83c4757005dd6c03e7333bba0b581cf266e720af088bd2247e: Status 404 returned error can't find the container with id 312b52ca954ecd83c4757005dd6c03e7333bba0b581cf266e720af088bd2247e
Jul 01 14:39:48.197663 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.197633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljxh4" event={"ID":"dfe1e7e8-3db4-4a7b-b138-27044da30c8d","Type":"ContainerStarted","Data":"312b52ca954ecd83c4757005dd6c03e7333bba0b581cf266e720af088bd2247e"}
Jul 01 14:39:48.198625 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.198605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xl6ft" event={"ID":"7c48eacc-cd11-4893-94ef-d8368f954658","Type":"ContainerStarted","Data":"f6fb18ccb65dd59bd0fb515d2445fb602ece5203a2c76420264b7a9d1cb2f308"}
Jul 01 14:39:48.533003 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.532920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:48.533492 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:48.533095    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:48.533492 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:48.533187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls podName:f3bec9de-cb72-4480-a078-60a9ab73b889 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:04.533166241 +0000 UTC m=+83.059450226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpkp9" (UID: "f3bec9de-cb72-4480-a078-60a9ab73b889") : secret "insights-runtime-extractor-tls" not found
Jul 01 14:39:48.533898 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.533709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:48.536992 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:48.534486    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3bec9de-cb72-4480-a078-60a9ab73b889-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:39:49.240740 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:49.240700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:39:49.240969 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:49.240904    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 14:39:49.241046 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:39:49.241007    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls podName:e4fa0250-95e6-4f97-9688-6c72a2910812 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:21.240985878 +0000 UTC m=+99.767269875 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls") pod "dns-default-g6n5t" (UID: "e4fa0250-95e6-4f97-9688-6c72a2910812") : secret "dns-default-metrics-tls" not found
Jul 01 14:39:52.213276 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:52.213226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljxh4" event={"ID":"dfe1e7e8-3db4-4a7b-b138-27044da30c8d","Type":"ContainerStarted","Data":"1dc96ade683ee2daf38caabb6255da25493fc312881c4e73027aee6f74ee04c0"}
Jul 01 14:39:52.213719 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:52.213368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:39:52.252283 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:52.252225    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ljxh4" podStartSLOduration=66.981405985 podStartE2EDuration="1m10.252204859s" totalImagesPullingTime="3.270798874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:52.250273233 +0000 UTC m=+70.776557238" watchObservedRunningTime="2026-07-01 14:39:52.252204859 +0000 UTC m=+70.778488862"
Jul 01 14:39:53.580978 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:53.580939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:53.583374 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:53.583344    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"image-registry-57db6cfdbc-sp9r4\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") " pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:53.678648 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:53.678619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9s77\""
Jul 01 14:39:53.686543 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:53.686523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:53.815127 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:53.815096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57db6cfdbc-sp9r4"]
Jul 01 14:39:53.818109 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:53.818082    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3be1174_0f05_4c7f_a7ca_4669b3b5467e.slice/crio-768f8c69048100fcdd57044f0de218e5e6f86aa0ab07be750fa82f4f4482f1ed WatchSource:0}: Error finding container 768f8c69048100fcdd57044f0de218e5e6f86aa0ab07be750fa82f4f4482f1ed: Status 404 returned error can't find the container with id 768f8c69048100fcdd57044f0de218e5e6f86aa0ab07be750fa82f4f4482f1ed
Jul 01 14:39:54.221599 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:54.221565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" event={"ID":"f3be1174-0f05-4c7f-a7ca-4669b3b5467e","Type":"ContainerStarted","Data":"69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331"}
Jul 01 14:39:54.221801 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:54.221624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" event={"ID":"f3be1174-0f05-4c7f-a7ca-4669b3b5467e","Type":"ContainerStarted","Data":"768f8c69048100fcdd57044f0de218e5e6f86aa0ab07be750fa82f4f4482f1ed"}
Jul 01 14:39:54.221801 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:54.221672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:39:54.223150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:54.223127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xl6ft" event={"ID":"7c48eacc-cd11-4893-94ef-d8368f954658","Type":"ContainerStarted","Data":"80ef5c14df4cd03559a7c3393578cd3a87591dfa9cc4ad908d6e7b7d6bc738e1"}
Jul 01 14:39:54.242164 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:54.242115    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" podStartSLOduration=17.242101495 podStartE2EDuration="17.242101495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:54.24151221 +0000 UTC m=+72.767796216" watchObservedRunningTime="2026-07-01 14:39:54.242101495 +0000 UTC m=+72.768385498"
Jul 01 14:39:54.259632 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:54.259588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xl6ft" podStartSLOduration=67.224915852 podStartE2EDuration="1m12.259573742s" totalImagesPullingTime="5.03465789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:39:54.258883168 +0000 UTC m=+72.785167173" watchObservedRunningTime="2026-07-01 14:39:54.259573742 +0000 UTC m=+72.785857745"
Jul 01 14:39:59.330226 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.330186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:59.330634 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.330258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:59.330634 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.330282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:59.330868 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.330847    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-service-ca-bundle\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:59.332564 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.332541    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2242c5d-0e33-4904-bd89-024c85303ff3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-s4fnf\" (UID: \"e2242c5d-0e33-4904-bd89-024c85303ff3\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:59.332707 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.332686    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb-metrics-certs\") pod \"router-default-674f7cb5-nhjq7\" (UID: \"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb\") " pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:59.503277 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.503249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vtgkn\""
Jul 01 14:39:59.510895 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.510870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"
Jul 01 14:39:59.547055 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.547020    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5mkmz\""
Jul 01 14:39:59.555249 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.555222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:39:59.652148 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.652119    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf"]
Jul 01 14:39:59.659894 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:59.659867    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2242c5d_0e33_4904_bd89_024c85303ff3.slice/crio-a90d817bf9657e1517aad721a5d38c302d3216edf087ef252056ab0a14866516 WatchSource:0}: Error finding container a90d817bf9657e1517aad721a5d38c302d3216edf087ef252056ab0a14866516: Status 404 returned error can't find the container with id a90d817bf9657e1517aad721a5d38c302d3216edf087ef252056ab0a14866516
Jul 01 14:39:59.695059 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:39:59.695030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674f7cb5-nhjq7"]
Jul 01 14:39:59.697770 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:39:59.697742    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ba0f3b_4fa8_43e1_8091_861ebf8eacdb.slice/crio-bdb58d8f9fea9939a1299c9049c7a0a30d576cff22d66c73407fcea36350efd0 WatchSource:0}: Error finding container bdb58d8f9fea9939a1299c9049c7a0a30d576cff22d66c73407fcea36350efd0: Status 404 returned error can't find the container with id bdb58d8f9fea9939a1299c9049c7a0a30d576cff22d66c73407fcea36350efd0
Jul 01 14:40:00.242284 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:00.242247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674f7cb5-nhjq7" event={"ID":"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb","Type":"ContainerStarted","Data":"5a139c20b22e43d3a87426e1dcc327b8f45cca58a2396ef944c7eaf1b87e78a4"}
Jul 01 14:40:00.242495 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:00.242327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674f7cb5-nhjq7" event={"ID":"b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb","Type":"ContainerStarted","Data":"bdb58d8f9fea9939a1299c9049c7a0a30d576cff22d66c73407fcea36350efd0"}
Jul 01 14:40:00.243508 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:00.243481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf" event={"ID":"e2242c5d-0e33-4904-bd89-024c85303ff3","Type":"ContainerStarted","Data":"a90d817bf9657e1517aad721a5d38c302d3216edf087ef252056ab0a14866516"}
Jul 01 14:40:00.267002 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:00.266852    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-674f7cb5-nhjq7" podStartSLOduration=33.266831221 podStartE2EDuration="33.266831221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:00.266437625 +0000 UTC m=+78.792721629" watchObservedRunningTime="2026-07-01 14:40:00.266831221 +0000 UTC m=+78.793115225"
Jul 01 14:40:00.555760 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:00.555679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:40:00.559412 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:00.559387    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:40:01.247557 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:01.247516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:40:01.249021 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:01.248998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-674f7cb5-nhjq7"
Jul 01 14:40:02.251764 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:02.251727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf" event={"ID":"e2242c5d-0e33-4904-bd89-024c85303ff3","Type":"ContainerStarted","Data":"cf48bad7e45bf7159558ba22013a5c153d85510f58049c9f2b5337ed1ccabf84"}
Jul 01 14:40:03.366412 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:03.366354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:40:03.368905 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:03.368877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-cng95\" (UID: \"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:40:03.538918 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:03.538887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c2qqz\""
Jul 01 14:40:03.547215 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:03.547189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"
Jul 01 14:40:03.682132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:03.682077    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-s4fnf" podStartSLOduration=35.028047971 podStartE2EDuration="36.68205947s" totalImagesPullingTime="1.654011499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:02.277165626 +0000 UTC m=+80.803449640" watchObservedRunningTime="2026-07-01 14:40:03.68205947 +0000 UTC m=+82.208343475"
Jul 01 14:40:03.682793 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:03.682774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95"]
Jul 01 14:40:04.258724 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:04.258615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95" event={"ID":"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac","Type":"ContainerStarted","Data":"cf08b839d815b5a377e69d59a0eafbdf48aba67f8fb9c7ffa4f37aa74f0bdb12"}
Jul 01 14:40:04.576558 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:04.576463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:40:04.578858 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:04.578833    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3bec9de-cb72-4480-a078-60a9ab73b889-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpkp9\" (UID: \"f3bec9de-cb72-4480-a078-60a9ab73b889\") " pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:40:04.613272 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:04.613244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g767b\""
Jul 01 14:40:04.621269 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:04.621246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpkp9"
Jul 01 14:40:04.769200 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:04.769165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpkp9"]
Jul 01 14:40:04.772002 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:04.771971    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bec9de_cb72_4480_a078_60a9ab73b889.slice/crio-bc708a586404c114ced48337a808700d13e87110e765975e7e387f22aee850eb WatchSource:0}: Error finding container bc708a586404c114ced48337a808700d13e87110e765975e7e387f22aee850eb: Status 404 returned error can't find the container with id bc708a586404c114ced48337a808700d13e87110e765975e7e387f22aee850eb
Jul 01 14:40:05.265009 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:05.264895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpkp9" event={"ID":"f3bec9de-cb72-4480-a078-60a9ab73b889","Type":"ContainerStarted","Data":"bc708a586404c114ced48337a808700d13e87110e765975e7e387f22aee850eb"}
Jul 01 14:40:07.273834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:07.273738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95" event={"ID":"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac","Type":"ContainerStarted","Data":"4b4f2cdc0642b12d6d298994932a4e7e702f3cc6820ae00344d90f80a2f434fc"}
Jul 01 14:40:07.273834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:07.273815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95" event={"ID":"c6359c7a-a0ae-4ee6-9ff5-cb82fece61ac","Type":"ContainerStarted","Data":"29c3f3ea4dc7e67b911d28de5599ebecdc2e195b25a9099ad82affc7e93b0d7d"}
Jul 01 14:40:07.301541 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:07.301483    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-cng95" podStartSLOduration=33.826610819 podStartE2EDuration="36.30146084s" totalImagesPullingTime="2.474850021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:07.300402784 +0000 UTC m=+85.826686813" watchObservedRunningTime="2026-07-01 14:40:07.30146084 +0000 UTC m=+85.827744846"
Jul 01 14:40:09.044693 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.044657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57db6cfdbc-sp9r4"]
Jul 01 14:40:09.049215 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.049186    2579 patch_prober.go:28] interesting pod/image-registry-57db6cfdbc-sp9r4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 14:40:09.049385 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.049237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" podUID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:40:09.130611 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.130561    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"]
Jul 01 14:40:09.133695 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.133669    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-rmtc6"]
Jul 01 14:40:09.133869 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.133795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:09.136200 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.136179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7x9gf\""
Jul 01 14:40:09.136484 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.136465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 01 14:40:09.136756 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.136741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:09.139026 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.139005    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 14:40:09.139532 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.139508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rj94r\""
Jul 01 14:40:09.139866 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.139846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 14:40:09.144036 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.143966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"]
Jul 01 14:40:09.145351 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.145330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-rmtc6"]
Jul 01 14:40:09.281796 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.281725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpkp9" event={"ID":"f3bec9de-cb72-4480-a078-60a9ab73b889","Type":"ContainerStarted","Data":"275b93e4b90f1da1684607cba0cdc6f385504f4b3a4ddc630b1cbf8caeffeca4"}
Jul 01 14:40:09.317353 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.317328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlf8z\" (UniqueName: \"kubernetes.io/projected/7b6625e6-3a8f-4859-b5aa-256c707589d2-kube-api-access-nlf8z\") pod \"downloads-5499b6fb8f-rmtc6\" (UID: \"7b6625e6-3a8f-4859-b5aa-256c707589d2\") " pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:09.317476 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.317410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b1aa3403-9b84-48e0-9d3b-fc674aa4af55-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-vsds7\" (UID: \"b1aa3403-9b84-48e0-9d3b-fc674aa4af55\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:09.418804 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.418757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b1aa3403-9b84-48e0-9d3b-fc674aa4af55-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-vsds7\" (UID: \"b1aa3403-9b84-48e0-9d3b-fc674aa4af55\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:09.418990 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.418859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlf8z\" (UniqueName: \"kubernetes.io/projected/7b6625e6-3a8f-4859-b5aa-256c707589d2-kube-api-access-nlf8z\") pod \"downloads-5499b6fb8f-rmtc6\" (UID: \"7b6625e6-3a8f-4859-b5aa-256c707589d2\") " pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:09.421265 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.421245    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b1aa3403-9b84-48e0-9d3b-fc674aa4af55-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-vsds7\" (UID: \"b1aa3403-9b84-48e0-9d3b-fc674aa4af55\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:09.430360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.430331    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlf8z\" (UniqueName: \"kubernetes.io/projected/7b6625e6-3a8f-4859-b5aa-256c707589d2-kube-api-access-nlf8z\") pod \"downloads-5499b6fb8f-rmtc6\" (UID: \"7b6625e6-3a8f-4859-b5aa-256c707589d2\") " pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:09.447220 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.447188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:09.454413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.454387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:09.607970 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.607602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"]
Jul 01 14:40:09.610514 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:09.610487    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1aa3403_9b84_48e0_9d3b_fc674aa4af55.slice/crio-360a1686119ad593a711eb146d9c1881500920837efe5dd98acf0e64fd199068 WatchSource:0}: Error finding container 360a1686119ad593a711eb146d9c1881500920837efe5dd98acf0e64fd199068: Status 404 returned error can't find the container with id 360a1686119ad593a711eb146d9c1881500920837efe5dd98acf0e64fd199068
Jul 01 14:40:09.625063 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:09.625029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-rmtc6"]
Jul 01 14:40:09.629146 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:09.629119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6625e6_3a8f_4859_b5aa_256c707589d2.slice/crio-34cfdfff286b096f382e67105dedc832798aef56e2675fe45c82f0c4513cd795 WatchSource:0}: Error finding container 34cfdfff286b096f382e67105dedc832798aef56e2675fe45c82f0c4513cd795: Status 404 returned error can't find the container with id 34cfdfff286b096f382e67105dedc832798aef56e2675fe45c82f0c4513cd795
Jul 01 14:40:10.288063 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:10.287987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-rmtc6" event={"ID":"7b6625e6-3a8f-4859-b5aa-256c707589d2","Type":"ContainerStarted","Data":"34cfdfff286b096f382e67105dedc832798aef56e2675fe45c82f0c4513cd795"}
Jul 01 14:40:10.289657 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:10.289626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7" event={"ID":"b1aa3403-9b84-48e0-9d3b-fc674aa4af55","Type":"ContainerStarted","Data":"360a1686119ad593a711eb146d9c1881500920837efe5dd98acf0e64fd199068"}
Jul 01 14:40:10.291707 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:10.291682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpkp9" event={"ID":"f3bec9de-cb72-4480-a078-60a9ab73b889","Type":"ContainerStarted","Data":"33fdb9df929f5c209460028dabc0b9af502efae10e16f883855cff8326ee5a05"}
Jul 01 14:40:13.310731 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.310688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpkp9" event={"ID":"f3bec9de-cb72-4480-a078-60a9ab73b889","Type":"ContainerStarted","Data":"1f30a96375246d8475706e65adaef83a5d8a50ee90eae7f0fc98e4276dd80ff7"}
Jul 01 14:40:13.312851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.312817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7" event={"ID":"b1aa3403-9b84-48e0-9d3b-fc674aa4af55","Type":"ContainerStarted","Data":"9dab216843f694589e2b5d28a33354bd404be45f41d09fbfac0cb98916ba04e2"}
Jul 01 14:40:13.313048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.313024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:13.318457 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.318435    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7"
Jul 01 14:40:13.333528 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.333480    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mpkp9" podStartSLOduration=33.909096269 podStartE2EDuration="41.333464174s" totalImagesPullingTime="7.424367905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 14:39:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:13.332242394 +0000 UTC m=+91.858526456" watchObservedRunningTime="2026-07-01 14:40:13.333464174 +0000 UTC m=+91.859748211"
Jul 01 14:40:13.354700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.354653    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-vsds7" podStartSLOduration=1.714192868 podStartE2EDuration="4.354636706s" totalImagesPullingTime="2.640443838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:13.354110608 +0000 UTC m=+91.880394613" watchObservedRunningTime="2026-07-01 14:40:13.354636706 +0000 UTC m=+91.880920710"
Jul 01 14:40:13.951512 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.951481    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-td7l8"]
Jul 01 14:40:13.955999 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.955966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:13.958697 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.958677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 14:40:13.959834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.959811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8b7kw\""
Jul 01 14:40:13.959955 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.959910    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 14:40:13.960114 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.960094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 14:40:13.969951 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:13.969929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-td7l8"]
Jul 01 14:40:14.061640 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.061603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8e7846f-bae0-4490-bc43-cebd44e2ccda-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.061640 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.061641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e7846f-bae0-4490-bc43-cebd44e2ccda-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.061891 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.061660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gppx5\" (UniqueName: \"kubernetes.io/projected/d8e7846f-bae0-4490-bc43-cebd44e2ccda-kube-api-access-gppx5\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.061891 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.061750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e7846f-bae0-4490-bc43-cebd44e2ccda-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.162550 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.162516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8e7846f-bae0-4490-bc43-cebd44e2ccda-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.162550 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.162551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e7846f-bae0-4490-bc43-cebd44e2ccda-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.162808 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.162569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gppx5\" (UniqueName: \"kubernetes.io/projected/d8e7846f-bae0-4490-bc43-cebd44e2ccda-kube-api-access-gppx5\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.162808 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.162609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e7846f-bae0-4490-bc43-cebd44e2ccda-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.163394 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.163355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8e7846f-bae0-4490-bc43-cebd44e2ccda-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.165250 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.165227    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8e7846f-bae0-4490-bc43-cebd44e2ccda-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.165503 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.165478    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8e7846f-bae0-4490-bc43-cebd44e2ccda-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.172518 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.172494    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gppx5\" (UniqueName: \"kubernetes.io/projected/d8e7846f-bae0-4490-bc43-cebd44e2ccda-kube-api-access-gppx5\") pod \"prometheus-operator-7b7d896497-td7l8\" (UID: \"d8e7846f-bae0-4490-bc43-cebd44e2ccda\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.267028 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.266944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8"
Jul 01 14:40:14.427709 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:14.427675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-td7l8"]
Jul 01 14:40:14.431291 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:14.431259    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e7846f_bae0_4490_bc43_cebd44e2ccda.slice/crio-04306ebc25d872f89d8c90b858ea5e137d52f93859bcabd639502d80bf723b8f WatchSource:0}: Error finding container 04306ebc25d872f89d8c90b858ea5e137d52f93859bcabd639502d80bf723b8f: Status 404 returned error can't find the container with id 04306ebc25d872f89d8c90b858ea5e137d52f93859bcabd639502d80bf723b8f
Jul 01 14:40:15.323081 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:15.323039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8" event={"ID":"d8e7846f-bae0-4490-bc43-cebd44e2ccda","Type":"ContainerStarted","Data":"04306ebc25d872f89d8c90b858ea5e137d52f93859bcabd639502d80bf723b8f"}
Jul 01 14:40:16.330073 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:16.330041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8" event={"ID":"d8e7846f-bae0-4490-bc43-cebd44e2ccda","Type":"ContainerStarted","Data":"8d056ae3b94ad35dcb96405f2b4bebc99109341c09bf7d206547ec07099fc803"}
Jul 01 14:40:16.330502 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:16.330096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8" event={"ID":"d8e7846f-bae0-4490-bc43-cebd44e2ccda","Type":"ContainerStarted","Data":"bb762860902a3d49764026832c04bfae788f8125b85c415521d4026fe2ff437c"}
Jul 01 14:40:16.353852 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:16.353796    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-td7l8" podStartSLOduration=1.974092169 podStartE2EDuration="3.353780172s" totalImagesPullingTime="1.379688003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:16.352323857 +0000 UTC m=+94.878607884" watchObservedRunningTime="2026-07-01 14:40:16.353780172 +0000 UTC m=+94.880064175"
Jul 01 14:40:18.332876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.332835    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"]
Jul 01 14:40:18.336456 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.336427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.339043 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.339019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 14:40:18.339182 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.339158    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 14:40:18.339943 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.339926    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4bksf\""
Jul 01 14:40:18.348679 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.348657    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"]
Jul 01 14:40:18.361267 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.361244    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l9q4n"]
Jul 01 14:40:18.364545 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.364527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.367032 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.367009    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 14:40:18.367140 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.367108    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 14:40:18.367349 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.367329    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 14:40:18.367619 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.367603    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jkt8g\""
Jul 01 14:40:18.398792 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.398756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.398957 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.398839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59ad5a3-db43-4872-99ba-ea96be6713f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.398957 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.398908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnlv\" (UniqueName: \"kubernetes.io/projected/dc82c488-6538-464d-947e-24f2151a3836-kube-api-access-sbnlv\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.398957 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.398939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-sys\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.398966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8lf\" (UniqueName: \"kubernetes.io/projected/d59ad5a3-db43-4872-99ba-ea96be6713f5-kube-api-access-gb8lf\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.399129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.398997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59ad5a3-db43-4872-99ba-ea96be6713f5-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.399129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-root\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399269 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399269 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-tls\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399269 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-wtmp\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399447 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-textfile\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399447 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc82c488-6538-464d-947e-24f2151a3836-metrics-client-ca\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.399447 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.399384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d59ad5a3-db43-4872-99ba-ea96be6713f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.500371 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59ad5a3-db43-4872-99ba-ea96be6713f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.500556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnlv\" (UniqueName: \"kubernetes.io/projected/dc82c488-6538-464d-947e-24f2151a3836-kube-api-access-sbnlv\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-sys\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8lf\" (UniqueName: \"kubernetes.io/projected/d59ad5a3-db43-4872-99ba-ea96be6713f5-kube-api-access-gb8lf\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.500556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59ad5a3-db43-4872-99ba-ea96be6713f5-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.500556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-root\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500537    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-sys\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-tls\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-wtmp\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-textfile\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-root\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc82c488-6538-464d-947e-24f2151a3836-metrics-client-ca\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d59ad5a3-db43-4872-99ba-ea96be6713f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.500933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.500825    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-wtmp\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.502571 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.501913    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-textfile\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.502571 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.502395    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc82c488-6538-464d-947e-24f2151a3836-metrics-client-ca\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.502571 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.502449    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59ad5a3-db43-4872-99ba-ea96be6713f5-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.502852 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.502829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.507547 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.504032    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.507547 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.504392    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59ad5a3-db43-4872-99ba-ea96be6713f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.507547 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.506284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d59ad5a3-db43-4872-99ba-ea96be6713f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.508294 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.508271    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc82c488-6538-464d-947e-24f2151a3836-node-exporter-tls\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.510839 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.510816    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8lf\" (UniqueName: \"kubernetes.io/projected/d59ad5a3-db43-4872-99ba-ea96be6713f5-kube-api-access-gb8lf\") pod \"openshift-state-metrics-57f99c8b49-fkvt2\" (UID: \"d59ad5a3-db43-4872-99ba-ea96be6713f5\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.511012 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.510990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnlv\" (UniqueName: \"kubernetes.io/projected/dc82c488-6538-464d-947e-24f2151a3836-kube-api-access-sbnlv\") pod \"node-exporter-l9q4n\" (UID: \"dc82c488-6538-464d-947e-24f2151a3836\") " pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.647469 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.647434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"
Jul 01 14:40:18.675917 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.675877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9q4n"
Jul 01 14:40:18.692024 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:18.691997    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc82c488_6538_464d_947e_24f2151a3836.slice/crio-6af409eafe7802b0d6af32a7fd40ff17f541f72cc9884b3b15bada0be8e6d34d WatchSource:0}: Error finding container 6af409eafe7802b0d6af32a7fd40ff17f541f72cc9884b3b15bada0be8e6d34d: Status 404 returned error can't find the container with id 6af409eafe7802b0d6af32a7fd40ff17f541f72cc9884b3b15bada0be8e6d34d
Jul 01 14:40:18.801816 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:18.801683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2"]
Jul 01 14:40:18.804285 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:18.804254    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59ad5a3_db43_4872_99ba_ea96be6713f5.slice/crio-92c866b9343bf12203bc22789a276c79559394d322ca684916bc6702bce8e7fa WatchSource:0}: Error finding container 92c866b9343bf12203bc22789a276c79559394d322ca684916bc6702bce8e7fa: Status 404 returned error can't find the container with id 92c866b9343bf12203bc22789a276c79559394d322ca684916bc6702bce8e7fa
Jul 01 14:40:19.051726 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.051664    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:40:19.344942 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.344821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2" event={"ID":"d59ad5a3-db43-4872-99ba-ea96be6713f5","Type":"ContainerStarted","Data":"2dbf267e60283e32e89af0a1719769723b390d5395b8270a9fd3b2aa954dcdc3"}
Jul 01 14:40:19.344942 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.344906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2" event={"ID":"d59ad5a3-db43-4872-99ba-ea96be6713f5","Type":"ContainerStarted","Data":"045512ff0502ea95085530f9a3e8d71a2a6f1eb0cb50a600048e76abeddfd135"}
Jul 01 14:40:19.344942 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.344927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2" event={"ID":"d59ad5a3-db43-4872-99ba-ea96be6713f5","Type":"ContainerStarted","Data":"92c866b9343bf12203bc22789a276c79559394d322ca684916bc6702bce8e7fa"}
Jul 01 14:40:19.346277 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.346238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9q4n" event={"ID":"dc82c488-6538-464d-947e-24f2151a3836","Type":"ContainerStarted","Data":"6af409eafe7802b0d6af32a7fd40ff17f541f72cc9884b3b15bada0be8e6d34d"}
Jul 01 14:40:19.476445 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.475076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:40:19.479767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.479740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.482462 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.482438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 14:40:19.482570 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.482463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lg8xj\""
Jul 01 14:40:19.482623 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.482599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 14:40:19.482767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.482723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 14:40:19.482767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.482759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 14:40:19.482894 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.482813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 14:40:19.483217 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.483069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 14:40:19.483683 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.483396    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 14:40:19.483683 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.483504    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 14:40:19.485549 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.485527    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 14:40:19.499922 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.499890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:40:19.612861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.612778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.612861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.612816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.612861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.612845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.612876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.612982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffkb\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-kube-api-access-vffkb\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-web-config\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-config-out\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613390 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.613575 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.613421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.713804 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-config-out\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.713980 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.713980 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.713980 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.713980 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.713980 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.713980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffkb\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-kube-api-access-vffkb\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.714256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-web-config\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.716244 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.714966    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.716244 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:40:19.715089    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 01 14:40:19.716244 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:40:19.715149    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls podName:e283ab02-2930-4626-9559-8aa25e3032e7 nodeName:}" failed. No retries permitted until 2026-07-01 14:40:20.215132075 +0000 UTC m=+98.741416071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7") : secret "alertmanager-main-tls" not found
Jul 01 14:40:19.716244 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.715209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.716244 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.715647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.717047 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.717022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.717244 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.717221    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-config-out\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.717460 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.717438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.717832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.717810    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.718230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.718210    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.718917 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.718878    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.719340 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.719321    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.719422 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.719350    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-web-config\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:19.729411 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:19.729386    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffkb\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-kube-api-access-vffkb\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:20.216614 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:20.216572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:20.219350 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:20.219317    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:20.394235 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:20.394193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:40:21.323232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:21.323195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:40:21.326138 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:21.326109    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4fa0250-95e6-4f97-9688-6c72a2910812-metrics-tls\") pod \"dns-default-g6n5t\" (UID: \"e4fa0250-95e6-4f97-9688-6c72a2910812\") " pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:40:21.596105 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:21.596022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4bfzd\""
Jul 01 14:40:21.604363 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:21.604336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:40:22.832661 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.832628    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85459649d6-n7jdh"]
Jul 01 14:40:22.836089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.836068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.838749 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.838721    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 01 14:40:22.838749 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.838739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zx92h\""
Jul 01 14:40:22.839854 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.839722    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 14:40:22.839854 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.839753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 01 14:40:22.839854 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.839756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 01 14:40:22.839854 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.839815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c5dr1v8amccvl\""
Jul 01 14:40:22.854627 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.854606    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85459649d6-n7jdh"]
Jul 01 14:40:22.935321 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-secret-metrics-server-client-certs\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.935480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-secret-metrics-server-tls\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.935480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-client-ca-bundle\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.935480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.935480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcl6\" (UniqueName: \"kubernetes.io/projected/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-kube-api-access-pbcl6\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.935645 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-metrics-server-audit-profiles\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:22.935645 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:22.935544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-audit-log\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036034 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-secret-metrics-server-client-certs\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-secret-metrics-server-tls\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-client-ca-bundle\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcl6\" (UniqueName: \"kubernetes.io/projected/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-kube-api-access-pbcl6\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-metrics-server-audit-profiles\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036496 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-audit-log\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.036575 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.036555    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-audit-log\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.037255 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.037235    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.037616 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.037591    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-metrics-server-audit-profiles\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.038939 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.038916    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-secret-metrics-server-client-certs\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.039034 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.038961    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-secret-metrics-server-tls\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.039576 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.039543    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-client-ca-bundle\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.050456 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.050408    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcl6\" (UniqueName: \"kubernetes.io/projected/f8c25f9b-945f-4acb-a4f5-523f0d2fb0de-kube-api-access-pbcl6\") pod \"metrics-server-85459649d6-n7jdh\" (UID: \"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de\") " pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.148058 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.148019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:23.178051 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.178011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"]
Jul 01 14:40:23.181379 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.181349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.185153 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.185125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mmnt4\""
Jul 01 14:40:23.185389 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.185374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 01 14:40:23.198083 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.197903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"]
Jul 01 14:40:23.220422 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.220390    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ljxh4"
Jul 01 14:40:23.238243 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.238193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbr4\" (UniqueName: \"kubernetes.io/projected/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-kube-api-access-xzbr4\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.238389 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.238353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.339146 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.339107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.339331 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.339194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbr4\" (UniqueName: \"kubernetes.io/projected/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-kube-api-access-xzbr4\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.339331 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:40:23.339277    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 01 14:40:23.339460 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:40:23.339371    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-monitoring-plugin-cert podName:ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc nodeName:}" failed. No retries permitted until 2026-07-01 14:40:23.839349725 +0000 UTC m=+102.365633726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-monitoring-plugin-cert") pod "monitoring-plugin-cd8656b8-48zcd" (UID: "ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc") : secret "monitoring-plugin-cert" not found
Jul 01 14:40:23.357393 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.357361    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbr4\" (UniqueName: \"kubernetes.io/projected/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-kube-api-access-xzbr4\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.844236 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.844200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:23.846943 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:23.846918    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-48zcd\" (UID: \"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:24.099405 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.099328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:24.766348 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.766291    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:40:24.817664 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.817438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:40:24.817664 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.817622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.820407 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.820379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 14:40:24.820543 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.820385    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-rou38k9vu717\""
Jul 01 14:40:24.820952 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.820921    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 14:40:24.821113 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 14:40:24.821226 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821151    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 14:40:24.821226 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821170    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 14:40:24.821765 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 14:40:24.821915 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821901    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 14:40:24.821985 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821957    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 14:40:24.822043 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.821996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9rnv4\""
Jul 01 14:40:24.822098 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.822074    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 14:40:24.822146 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.822106    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 14:40:24.822146 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.822128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 14:40:24.822233 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.822144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 14:40:24.823897 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.823874    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 14:40:24.829988 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.829966    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4wf\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-kube-api-access-9x4wf\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.856994 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.858252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.858252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.858252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.858252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.854973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956012 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.955983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956012 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4wf\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-kube-api-access-9x4wf\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.956402 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.957103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.957103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.957103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.957103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.957103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956840    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.957103 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.956896    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.959253 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.959210    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.959826    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.960027    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.960298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.962084    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.962131    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.962587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.962771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.962681    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.964445 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.963803    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.964646 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.964545    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-web-config\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.970835 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.970771    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.970963 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.970857    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.971029 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.970969    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.971419 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.971398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.972403 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.972349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.972403 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.972360    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config-out\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.973052 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.973027    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:24.977601 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:24.977576    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4wf\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-kube-api-access-9x4wf\") pod \"prometheus-k8s-0\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:25.130758 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:25.130673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:27.905570 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:27.904871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:40:27.913396 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:27.913348    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode283ab02_2930_4626_9559_8aa25e3032e7.slice/crio-31d18ac496b0d2d1d685142e45b093bd7704d68e3eff0dbe646b155ca8ab652e WatchSource:0}: Error finding container 31d18ac496b0d2d1d685142e45b093bd7704d68e3eff0dbe646b155ca8ab652e: Status 404 returned error can't find the container with id 31d18ac496b0d2d1d685142e45b093bd7704d68e3eff0dbe646b155ca8ab652e
Jul 01 14:40:27.957989 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:27.957958    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ca1efb_b7b0_43e6_8094_1f4dc17c104b.slice/crio-7d17ac94ea2843f0bb51aaa360f1748ecf959d5b7cdbfc05fd167c3475cf3cc1 WatchSource:0}: Error finding container 7d17ac94ea2843f0bb51aaa360f1748ecf959d5b7cdbfc05fd167c3475cf3cc1: Status 404 returned error can't find the container with id 7d17ac94ea2843f0bb51aaa360f1748ecf959d5b7cdbfc05fd167c3475cf3cc1
Jul 01 14:40:27.958120 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:27.958043    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:40:27.968990 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:27.968961    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85459649d6-n7jdh"]
Jul 01 14:40:27.972915 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:27.972887    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c25f9b_945f_4acb_a4f5_523f0d2fb0de.slice/crio-428ff92e175f4e50a6c2cc1f6a0b3c3b61530c8f00060f4a828e97075e7f4317 WatchSource:0}: Error finding container 428ff92e175f4e50a6c2cc1f6a0b3c3b61530c8f00060f4a828e97075e7f4317: Status 404 returned error can't find the container with id 428ff92e175f4e50a6c2cc1f6a0b3c3b61530c8f00060f4a828e97075e7f4317
Jul 01 14:40:28.108848 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.108815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"]
Jul 01 14:40:28.111752 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.111726    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g6n5t"]
Jul 01 14:40:28.112262 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:28.112232    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8fd0f3_7b89_4e23_a6c1_797c6cb337cc.slice/crio-03cb6472ac1f0bc6f273fd9d0a646df74c2bd6ee8e9d63d0ca259b6c4d9a7d66 WatchSource:0}: Error finding container 03cb6472ac1f0bc6f273fd9d0a646df74c2bd6ee8e9d63d0ca259b6c4d9a7d66: Status 404 returned error can't find the container with id 03cb6472ac1f0bc6f273fd9d0a646df74c2bd6ee8e9d63d0ca259b6c4d9a7d66
Jul 01 14:40:28.114357 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:28.114333    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fa0250_95e6_4f97_9688_6c72a2910812.slice/crio-0dbd8c3bde9f14747bf738275bfab6740fab57219e470f2a02ca52ea266b337b WatchSource:0}: Error finding container 0dbd8c3bde9f14747bf738275bfab6740fab57219e470f2a02ca52ea266b337b: Status 404 returned error can't find the container with id 0dbd8c3bde9f14747bf738275bfab6740fab57219e470f2a02ca52ea266b337b
Jul 01 14:40:28.384150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.384054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"31d18ac496b0d2d1d685142e45b093bd7704d68e3eff0dbe646b155ca8ab652e"}
Jul 01 14:40:28.385505 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.385470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh" event={"ID":"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de","Type":"ContainerStarted","Data":"428ff92e175f4e50a6c2cc1f6a0b3c3b61530c8f00060f4a828e97075e7f4317"}
Jul 01 14:40:28.386817 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.386787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g6n5t" event={"ID":"e4fa0250-95e6-4f97-9688-6c72a2910812","Type":"ContainerStarted","Data":"0dbd8c3bde9f14747bf738275bfab6740fab57219e470f2a02ca52ea266b337b"}
Jul 01 14:40:28.388530 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.388501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-rmtc6" event={"ID":"7b6625e6-3a8f-4859-b5aa-256c707589d2","Type":"ContainerStarted","Data":"3cd0e6dadc086795dff257d3644d5a91f2eedf5c11d0248ffeee4c8051676d1e"}
Jul 01 14:40:28.388744 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.388670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:28.390120 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.390095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"7d17ac94ea2843f0bb51aaa360f1748ecf959d5b7cdbfc05fd167c3475cf3cc1"}
Jul 01 14:40:28.391606 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.391584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd" event={"ID":"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc","Type":"ContainerStarted","Data":"03cb6472ac1f0bc6f273fd9d0a646df74c2bd6ee8e9d63d0ca259b6c4d9a7d66"}
Jul 01 14:40:28.393188 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.393164    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc82c488-6538-464d-947e-24f2151a3836" containerID="d80c5407b7194d6dfdb0c28f5a073e7321b6d173fd4c3297a9ffff206fb96799" exitCode=0
Jul 01 14:40:28.393280 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.393248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9q4n" event={"ID":"dc82c488-6538-464d-947e-24f2151a3836","Type":"ContainerDied","Data":"d80c5407b7194d6dfdb0c28f5a073e7321b6d173fd4c3297a9ffff206fb96799"}
Jul 01 14:40:28.395615 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.395592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2" event={"ID":"d59ad5a3-db43-4872-99ba-ea96be6713f5","Type":"ContainerStarted","Data":"33ed4e90f2ec81c03d709fdb269c059bb8abfdc91c43e288c6c9a3bdb35cd92a"}
Jul 01 14:40:28.401552 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.401532    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-rmtc6"
Jul 01 14:40:28.409088 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.409037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-rmtc6" podStartSLOduration=1.357735748 podStartE2EDuration="19.409019183s" totalImagesPullingTime="18.051283435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:28.407070196 +0000 UTC m=+106.933354200" watchObservedRunningTime="2026-07-01 14:40:28.409019183 +0000 UTC m=+106.935303186"
Jul 01 14:40:28.427942 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:28.427885    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-fkvt2" podStartSLOduration=1.696660451 podStartE2EDuration="10.427863092s" totalImagesPullingTime="8.731202641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:28.425988113 +0000 UTC m=+106.952272118" watchObservedRunningTime="2026-07-01 14:40:28.427863092 +0000 UTC m=+106.954147098"
Jul 01 14:40:29.411556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:29.410818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9q4n" event={"ID":"dc82c488-6538-464d-947e-24f2151a3836","Type":"ContainerStarted","Data":"9076d47a3e4bdafbf39144fddff4b8f7bfc92cc6f91bcb6ad9fc7b6e2102d863"}
Jul 01 14:40:29.411556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:29.410919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9q4n" event={"ID":"dc82c488-6538-464d-947e-24f2151a3836","Type":"ContainerStarted","Data":"ba1b16284f8a4742a87bf3ef94b58201c06db4c8b1f171195995ac346003eabe"}
Jul 01 14:40:29.452208 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:29.451981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l9q4n" podStartSLOduration=2.384401542 podStartE2EDuration="11.451955695s" totalImagesPullingTime="8.986328049s" totalInitContainerRuntime="81.226104ms" isStatefulPod=true podCreationTimestamp="2026-07-01 14:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:29.43900936 +0000 UTC m=+107.965293365" watchObservedRunningTime="2026-07-01 14:40:29.451955695 +0000 UTC m=+107.978239702"
Jul 01 14:40:32.437649 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:32.437527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh" event={"ID":"f8c25f9b-945f-4acb-a4f5-523f0d2fb0de","Type":"ContainerStarted","Data":"094428b8a327fafc356fb16c5722ab56eb0a3d0faf89b181b3958d870d57cc05"}
Jul 01 14:40:33.444533 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.444488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g6n5t" event={"ID":"e4fa0250-95e6-4f97-9688-6c72a2910812","Type":"ContainerStarted","Data":"c1579d7d3bdcc9d3af3d05fd8acc9e1caba8f5e6a7a5ca2b436a95a18c8e1351"}
Jul 01 14:40:33.444996 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.444563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g6n5t" event={"ID":"e4fa0250-95e6-4f97-9688-6c72a2910812","Type":"ContainerStarted","Data":"ea30b05f1046eee9520bd500f9aeb1fd22322f31a9dc02ab8ea92ad2af7c4dad"}
Jul 01 14:40:33.445456 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.445429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:40:33.447548 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.447518    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7" exitCode=0
Jul 01 14:40:33.447653 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.447565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"}
Jul 01 14:40:33.449425 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.449393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd" event={"ID":"ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc","Type":"ContainerStarted","Data":"0d19756d5fe0b630f95f6248206898033ced67c503ef1d75567b6d62fc134f8f"}
Jul 01 14:40:33.450323 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.449876    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:33.451455 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.451427    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72" exitCode=0
Jul 01 14:40:33.451565 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.451499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72"}
Jul 01 14:40:33.457645 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.457624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd"
Jul 01 14:40:33.465101 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.465048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g6n5t" podStartSLOduration=72.309229548 podStartE2EDuration="1m16.465029131s" totalImagesPullingTime="4.155799583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:33.464436811 +0000 UTC m=+111.990720816" watchObservedRunningTime="2026-07-01 14:40:33.465029131 +0000 UTC m=+111.991313139"
Jul 01 14:40:33.486469 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.486408    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-48zcd" podStartSLOduration=5.9309520540000005 podStartE2EDuration="10.486373182s" totalImagesPullingTime="4.555421128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:33.485402544 +0000 UTC m=+112.011686549" watchObservedRunningTime="2026-07-01 14:40:33.486373182 +0000 UTC m=+112.012657187"
Jul 01 14:40:33.612805 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:33.612683    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh" podStartSLOduration=7.315507376 podStartE2EDuration="11.612661568s" totalImagesPullingTime="4.297154192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:33.610927951 +0000 UTC m=+112.137211956" watchObservedRunningTime="2026-07-01 14:40:33.612661568 +0000 UTC m=+112.138945577"
Jul 01 14:40:34.064881 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.064835    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" podUID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" containerName="registry" containerID="cri-o://69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331" gracePeriod=30
Jul 01 14:40:34.377774 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.377736    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:40:34.461147 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.460830    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" containerID="69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331" exitCode=0
Jul 01 14:40:34.461609 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.461400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4"
Jul 01 14:40:34.461998 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.461698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" event={"ID":"f3be1174-0f05-4c7f-a7ca-4669b3b5467e","Type":"ContainerDied","Data":"69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331"}
Jul 01 14:40:34.461998 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.461751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57db6cfdbc-sp9r4" event={"ID":"f3be1174-0f05-4c7f-a7ca-4669b3b5467e","Type":"ContainerDied","Data":"768f8c69048100fcdd57044f0de218e5e6f86aa0ab07be750fa82f4f4482f1ed"}
Jul 01 14:40:34.461998 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.461783    2579 scope.go:117] "RemoveContainer" containerID="69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331"
Jul 01 14:40:34.466479 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqqp4\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-kube-api-access-qqqp4\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.466479 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466516    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-bound-sa-token\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-image-registry-private-configuration\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466591    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-ca-trust-extracted\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-installation-pull-secrets\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-certificates\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.466700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-trusted-ca\") pod \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\" (UID: \"f3be1174-0f05-4c7f-a7ca-4669b3b5467e\") "
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.467685    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:40:34.469048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.468022    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:40:34.474974 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.474925    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-kube-api-access-qqqp4" (OuterVolumeSpecName: "kube-api-access-qqqp4") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "kube-api-access-qqqp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:40:34.475082 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.475013    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:40:34.475781 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.475737    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:40:34.475781 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.475757    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:40:34.476159 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.476094    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:40:34.477283 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.477145    2579 scope.go:117] "RemoveContainer" containerID="69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331"
Jul 01 14:40:34.478211 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:40:34.477871    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331\": container with ID starting with 69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331 not found: ID does not exist" containerID="69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331"
Jul 01 14:40:34.478211 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.477909    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331"} err="failed to get container status \"69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331\": rpc error: code = NotFound desc = could not find container \"69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331\": container with ID starting with 69748ebb6006ad6ebfeb002999d901f57ab2dcabfcfb089efa5c59bca8f3d331 not found: ID does not exist"
Jul 01 14:40:34.481636 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.481613    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f3be1174-0f05-4c7f-a7ca-4669b3b5467e" (UID: "f3be1174-0f05-4c7f-a7ca-4669b3b5467e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568273    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqp4\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-kube-api-access-qqqp4\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568329    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568344    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-bound-sa-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568358    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-image-registry-private-configuration\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568373    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-ca-trust-extracted\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568387    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-installation-pull-secrets\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568403    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-registry-certificates\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.569013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.568417    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3be1174-0f05-4c7f-a7ca-4669b3b5467e-trusted-ca\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:40:34.793562 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.793521    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57db6cfdbc-sp9r4"]
Jul 01 14:40:34.798585 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:34.798548    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57db6cfdbc-sp9r4"]
Jul 01 14:40:35.960167 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:35.959720    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" path="/var/lib/kubelet/pods/f3be1174-0f05-4c7f-a7ca-4669b3b5467e/volumes"
Jul 01 14:40:36.477199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:36.477135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18"}
Jul 01 14:40:36.477348 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:36.477216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc"}
Jul 01 14:40:36.477348 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:36.477256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f"}
Jul 01 14:40:36.477348 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:36.477291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55"}
Jul 01 14:40:37.489130 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:37.489087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9"}
Jul 01 14:40:38.496588 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:38.496554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"}
Jul 01 14:40:38.497026 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:38.496626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"}
Jul 01 14:40:39.503720 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:39.503686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerStarted","Data":"d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97"}
Jul 01 14:40:39.543915 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:39.543855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=11.549694622 podStartE2EDuration="20.543833931s" totalImagesPullingTime="8.965992075s" totalInitContainerRuntime="28.147234ms" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:39.542482217 +0000 UTC m=+118.068766220" watchObservedRunningTime="2026-07-01 14:40:39.543833931 +0000 UTC m=+118.070117938"
Jul 01 14:40:42.519673 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:42.519644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"}
Jul 01 14:40:42.520018 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:42.519731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"}
Jul 01 14:40:43.148918 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:43.148883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:43.149081 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:43.148944    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:40:43.527639 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:43.527567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"}
Jul 01 14:40:43.527639 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:43.527631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerStarted","Data":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"}
Jul 01 14:40:43.563583 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:43.563518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.762024339 podStartE2EDuration="19.563497148s" totalImagesPullingTime="12.768385728s" totalInitContainerRuntime="33.087081ms" isStatefulPod=false podCreationTimestamp="2026-07-01 14:40:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:43.561407861 +0000 UTC m=+122.087691883" watchObservedRunningTime="2026-07-01 14:40:43.563497148 +0000 UTC m=+122.089781153"
Jul 01 14:40:44.470787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:44.470758    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g6n5t"
Jul 01 14:40:45.131133 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:45.131092    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:40:49.554882 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:49.554840    2579 generic.go:356] "Generic (PLEG): container finished" podID="d359dafa-4332-46a3-ae6c-4d6197a474f6" containerID="49f3c09ad11a8725716c3bf082e9a54e4dd6852a3cf98170a1b5826955fb1984" exitCode=0
Jul 01 14:40:49.555333 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:49.554912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr" event={"ID":"d359dafa-4332-46a3-ae6c-4d6197a474f6","Type":"ContainerDied","Data":"49f3c09ad11a8725716c3bf082e9a54e4dd6852a3cf98170a1b5826955fb1984"}
Jul 01 14:40:49.555429 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:49.555410    2579 scope.go:117] "RemoveContainer" containerID="49f3c09ad11a8725716c3bf082e9a54e4dd6852a3cf98170a1b5826955fb1984"
Jul 01 14:40:50.560939 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:50.560905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-8dfkr" event={"ID":"d359dafa-4332-46a3-ae6c-4d6197a474f6","Type":"ContainerStarted","Data":"afb5832913eabd3e24e3aede714d4efad4c4a027f01cc7337c665587a77dcdf5"}
Jul 01 14:40:51.843248 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:51.843213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:40:51.845544 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:51.845518    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12300cae-4145-449e-9bac-bf1b507ab3da-metrics-certs\") pod \"network-metrics-daemon-vwzh5\" (UID: \"12300cae-4145-449e-9bac-bf1b507ab3da\") " pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:40:51.966059 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:51.966032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l5f8j\""
Jul 01 14:40:51.974293 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:51.974274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vwzh5"
Jul 01 14:40:52.123129 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:52.123105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vwzh5"]
Jul 01 14:40:52.125072 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:40:52.125048    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12300cae_4145_449e_9bac_bf1b507ab3da.slice/crio-1d96454eac95ed59ad6d1872f9b1be36427e8fd2cffdac9b28377d02c15b9f77 WatchSource:0}: Error finding container 1d96454eac95ed59ad6d1872f9b1be36427e8fd2cffdac9b28377d02c15b9f77: Status 404 returned error can't find the container with id 1d96454eac95ed59ad6d1872f9b1be36427e8fd2cffdac9b28377d02c15b9f77
Jul 01 14:40:52.570476 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:52.570441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vwzh5" event={"ID":"12300cae-4145-449e-9bac-bf1b507ab3da","Type":"ContainerStarted","Data":"1d96454eac95ed59ad6d1872f9b1be36427e8fd2cffdac9b28377d02c15b9f77"}
Jul 01 14:40:55.583915 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:55.583877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vwzh5" event={"ID":"12300cae-4145-449e-9bac-bf1b507ab3da","Type":"ContainerStarted","Data":"0b4362c90c13b9ce49a2d329eb53ef6c51cf399f519ca45b0c247b262789c068"}
Jul 01 14:40:55.584326 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:40:55.583934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vwzh5" event={"ID":"12300cae-4145-449e-9bac-bf1b507ab3da","Type":"ContainerStarted","Data":"77fb0ac69235282c31fbd157975c65480d4074a6885768eaae316b52f927bef8"}
Jul 01 14:41:02.613282 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:02.613248    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255" containerID="29801719e19bf5610362edda0111909be01d586d612a64bb5fafe9d46d5b85fc" exitCode=0
Jul 01 14:41:02.613677 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:02.613325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-l9tjh" event={"ID":"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255","Type":"ContainerDied","Data":"29801719e19bf5610362edda0111909be01d586d612a64bb5fafe9d46d5b85fc"}
Jul 01 14:41:02.613677 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:02.613652    2579 scope.go:117] "RemoveContainer" containerID="29801719e19bf5610362edda0111909be01d586d612a64bb5fafe9d46d5b85fc"
Jul 01 14:41:02.639806 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:02.639756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vwzh5" podStartSLOduration=138.017628127 podStartE2EDuration="2m20.639739599s" totalImagesPullingTime="2.622111472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:40:55.614341912 +0000 UTC m=+134.140625917" watchObservedRunningTime="2026-07-01 14:41:02.639739599 +0000 UTC m=+141.166023602"
Jul 01 14:41:03.155330 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:03.155277    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:41:03.161456 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:03.161427    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85459649d6-n7jdh"
Jul 01 14:41:03.619343 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:03.619227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-l9tjh" event={"ID":"c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255","Type":"ContainerStarted","Data":"a1c70fe4b763d885d77533e50bb549eb6d620dcfc5be1bd550d6f560b485110e"}
Jul 01 14:41:25.131824 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:25.131778    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:25.147556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:25.147533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:25.729693 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:25.729668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:38.840095 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840013    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:41:38.840591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840327    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="alertmanager" containerID="cri-o://22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55" gracePeriod=120
Jul 01 14:41:38.840591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840351    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-metric" containerID="cri-o://a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9" gracePeriod=120
Jul 01 14:41:38.840591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840377    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-web" containerID="cri-o://fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc" gracePeriod=120
Jul 01 14:41:38.840591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840401    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy" containerID="cri-o://672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18" gracePeriod=120
Jul 01 14:41:38.840591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840440    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="config-reloader" containerID="cri-o://dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f" gracePeriod=120
Jul 01 14:41:38.840591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:38.840446    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="prom-label-proxy" containerID="cri-o://d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97" gracePeriod=120
Jul 01 14:41:39.776529 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776492    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97" exitCode=0
Jul 01 14:41:39.776529 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776519    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9" exitCode=0
Jul 01 14:41:39.776529 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776527    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18" exitCode=0
Jul 01 14:41:39.776529 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776535    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f" exitCode=0
Jul 01 14:41:39.776809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776542    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55" exitCode=0
Jul 01 14:41:39.776809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97"}
Jul 01 14:41:39.776809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9"}
Jul 01 14:41:39.776809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18"}
Jul 01 14:41:39.776809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f"}
Jul 01 14:41:39.776809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:39.776701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55"}
Jul 01 14:41:40.093381 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.093353    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.244608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-cluster-tls-config\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-tls-assets\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-main-db\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vffkb\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-kube-api-access-vffkb\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-web-config\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-config-volume\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.244880 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-metrics-client-ca\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.245232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-config-out\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.245232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244916    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.245232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.245232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.244993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-trusted-ca-bundle\") pod \"e283ab02-2930-4626-9559-8aa25e3032e7\" (UID: \"e283ab02-2930-4626-9559-8aa25e3032e7\") "
Jul 01 14:41:40.245232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.245010    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:41:40.245524 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.245268    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-main-db\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.245674 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.245645    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:40.246021 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.245997    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:40.247498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.247416    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-kube-api-access-vffkb" (OuterVolumeSpecName: "kube-api-access-vffkb") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "kube-api-access-vffkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:41:40.247829 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.247768    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.248050 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.248007    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.248050 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.248037    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-config-out" (OuterVolumeSpecName: "config-out") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:41:40.248235 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.248158    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:41:40.248296 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.248237    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.248413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.248383    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.249194 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.249171    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-config-volume" (OuterVolumeSpecName: "config-volume") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.252609 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.252588    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.260051 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.260027    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-web-config" (OuterVolumeSpecName: "web-config") pod "e283ab02-2930-4626-9559-8aa25e3032e7" (UID: "e283ab02-2930-4626-9559-8aa25e3032e7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346502    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-web-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346551    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346562    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-config-volume\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346571    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-metrics-client-ca\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346581    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e283ab02-2930-4626-9559-8aa25e3032e7-config-out\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346590    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346600    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-main-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346609    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e283ab02-2930-4626-9559-8aa25e3032e7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346617    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-cluster-tls-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.346608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346626    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e283ab02-2930-4626-9559-8aa25e3032e7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.347024 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346634    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-tls-assets\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.347024 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.346643    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vffkb\" (UniqueName: \"kubernetes.io/projected/e283ab02-2930-4626-9559-8aa25e3032e7-kube-api-access-vffkb\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:40.784389 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.784357    2579 generic.go:356] "Generic (PLEG): container finished" podID="e283ab02-2930-4626-9559-8aa25e3032e7" containerID="fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc" exitCode=0
Jul 01 14:41:40.784552 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.784422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc"}
Jul 01 14:41:40.784552 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.784479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e283ab02-2930-4626-9559-8aa25e3032e7","Type":"ContainerDied","Data":"31d18ac496b0d2d1d685142e45b093bd7704d68e3eff0dbe646b155ca8ab652e"}
Jul 01 14:41:40.784552 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.784427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.784552 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.784520    2579 scope.go:117] "RemoveContainer" containerID="d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97"
Jul 01 14:41:40.793025 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.793006    2579 scope.go:117] "RemoveContainer" containerID="a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9"
Jul 01 14:41:40.800630 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.800611    2579 scope.go:117] "RemoveContainer" containerID="672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18"
Jul 01 14:41:40.807535 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.807512    2579 scope.go:117] "RemoveContainer" containerID="fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc"
Jul 01 14:41:40.814410 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.814387    2579 scope.go:117] "RemoveContainer" containerID="dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f"
Jul 01 14:41:40.815683 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.815667    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:41:40.821817 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.821800    2579 scope.go:117] "RemoveContainer" containerID="22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55"
Jul 01 14:41:40.823258 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.823232    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:41:40.828850 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.828826    2579 scope.go:117] "RemoveContainer" containerID="5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72"
Jul 01 14:41:40.836343 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.836327    2579 scope.go:117] "RemoveContainer" containerID="d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97"
Jul 01 14:41:40.836581 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.836563    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97\": container with ID starting with d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97 not found: ID does not exist" containerID="d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97"
Jul 01 14:41:40.836664 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.836591    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97"} err="failed to get container status \"d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97\": rpc error: code = NotFound desc = could not find container \"d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97\": container with ID starting with d873474b05b4cb1a334e8c174a7d6a334ad91b1b540b7d743fb4c4b1c5593e97 not found: ID does not exist"
Jul 01 14:41:40.836664 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.836618    2579 scope.go:117] "RemoveContainer" containerID="a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9"
Jul 01 14:41:40.836860 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.836841    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9\": container with ID starting with a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9 not found: ID does not exist" containerID="a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9"
Jul 01 14:41:40.836924 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.836869    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9"} err="failed to get container status \"a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9\": rpc error: code = NotFound desc = could not find container \"a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9\": container with ID starting with a8f6a9a9ba1c4c2ccaef4be2e3a7ac60b1f4daf5b95f092d42490195a7e21ae9 not found: ID does not exist"
Jul 01 14:41:40.836924 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.836892    2579 scope.go:117] "RemoveContainer" containerID="672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18"
Jul 01 14:41:40.837153 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.837128    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18\": container with ID starting with 672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18 not found: ID does not exist" containerID="672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18"
Jul 01 14:41:40.837194 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837159    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18"} err="failed to get container status \"672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18\": rpc error: code = NotFound desc = could not find container \"672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18\": container with ID starting with 672fe746ee09f1c5b40d1a830df4a4f8f29dbc3835a940d7ac4df3181e185c18 not found: ID does not exist"
Jul 01 14:41:40.837194 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837175    2579 scope.go:117] "RemoveContainer" containerID="fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc"
Jul 01 14:41:40.837430 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.837409    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc\": container with ID starting with fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc not found: ID does not exist" containerID="fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc"
Jul 01 14:41:40.837511 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837432    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc"} err="failed to get container status \"fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc\": rpc error: code = NotFound desc = could not find container \"fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc\": container with ID starting with fcfba28b261bc1a1683cadf28583dfc7f70273555783ea937567ed8d5f8e60dc not found: ID does not exist"
Jul 01 14:41:40.837511 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837447    2579 scope.go:117] "RemoveContainer" containerID="dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f"
Jul 01 14:41:40.837663 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.837644    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f\": container with ID starting with dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f not found: ID does not exist" containerID="dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f"
Jul 01 14:41:40.837699 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f"} err="failed to get container status \"dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f\": rpc error: code = NotFound desc = could not find container \"dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f\": container with ID starting with dbecf3489517ab48d639a80f0e69f8dfe3eaa47ce7f5d08ee479b6e6ee1f180f not found: ID does not exist"
Jul 01 14:41:40.837699 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837684    2579 scope.go:117] "RemoveContainer" containerID="22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55"
Jul 01 14:41:40.837900 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.837887    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55\": container with ID starting with 22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55 not found: ID does not exist" containerID="22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55"
Jul 01 14:41:40.837934 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837904    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55"} err="failed to get container status \"22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55\": rpc error: code = NotFound desc = could not find container \"22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55\": container with ID starting with 22769b8701844df30f1a9d02682c57d5915c40a21a90c3659d34aeb64d56cc55 not found: ID does not exist"
Jul 01 14:41:40.837934 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.837917    2579 scope.go:117] "RemoveContainer" containerID="5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72"
Jul 01 14:41:40.838128 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:40.838114    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72\": container with ID starting with 5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72 not found: ID does not exist" containerID="5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72"
Jul 01 14:41:40.838166 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.838130    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72"} err="failed to get container status \"5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72\": rpc error: code = NotFound desc = could not find container \"5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72\": container with ID starting with 5f2fbe7873db1e9a4ab917180a2057a2ba2e6332423b128cba14cbdc6e5fea72 not found: ID does not exist"
Jul 01 14:41:40.868130 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868103    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:41:40.868445 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868433    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-web"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868447    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-web"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868459    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-metric"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868465    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-metric"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868476    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="config-reloader"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868482    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="config-reloader"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868489    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" containerName="registry"
Jul 01 14:41:40.868498 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868494    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" containerName="registry"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868502    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868508    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868513    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="prom-label-proxy"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868520    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="prom-label-proxy"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868528    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="init-config-reloader"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868533    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="init-config-reloader"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868540    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="alertmanager"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868544    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="alertmanager"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868590    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="prom-label-proxy"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868598    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868605    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-metric"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868611    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3be1174-0f05-4c7f-a7ca-4669b3b5467e" containerName="registry"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868619    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="kube-rbac-proxy-web"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868624    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="config-reloader"
Jul 01 14:41:40.868700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.868630    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" containerName="alertmanager"
Jul 01 14:41:40.872420 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.872404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.876842 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.876813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 14:41:40.877036 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 14:41:40.877584 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 14:41:40.877584 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 14:41:40.877584 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lg8xj\""
Jul 01 14:41:40.877584 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 14:41:40.877584 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877531    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 14:41:40.877882 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 14:41:40.877882 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.877716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 14:41:40.885418 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.885399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 14:41:40.894858 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.894776    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:41:40.951959 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.951925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f3bec10-78d2-4de0-803e-a844b37c72cb-config-out\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.951959 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.951962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952138 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.951991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952138 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f3bec10-78d2-4de0-803e-a844b37c72cb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952327 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclh8\" (UniqueName: \"kubernetes.io/projected/1f3bec10-78d2-4de0-803e-a844b37c72cb-kube-api-access-tclh8\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952327 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3bec10-78d2-4de0-803e-a844b37c72cb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f3bec10-78d2-4de0-803e-a844b37c72cb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3bec10-78d2-4de0-803e-a844b37c72cb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:40.952591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:40.952449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-web-config\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.053838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclh8\" (UniqueName: \"kubernetes.io/projected/1f3bec10-78d2-4de0-803e-a844b37c72cb-kube-api-access-tclh8\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.053838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.053838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3bec10-78d2-4de0-803e-a844b37c72cb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f3bec10-78d2-4de0-803e-a844b37c72cb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3bec10-78d2-4de0-803e-a844b37c72cb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-web-config\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.053987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f3bec10-78d2-4de0-803e-a844b37c72cb-config-out\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f3bec10-78d2-4de0-803e-a844b37c72cb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054651 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054733 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054706    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3bec10-78d2-4de0-803e-a844b37c72cb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.054904 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.054878    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3bec10-78d2-4de0-803e-a844b37c72cb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.055073 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.055012    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f3bec10-78d2-4de0-803e-a844b37c72cb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057121 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057045    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f3bec10-78d2-4de0-803e-a844b37c72cb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057121 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057051    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057287 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057217    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057376 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057327    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057436 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057669 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057649    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057894 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057871    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.057933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.057877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f3bec10-78d2-4de0-803e-a844b37c72cb-web-config\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.058885 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.058869    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f3bec10-78d2-4de0-803e-a844b37c72cb-config-out\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.064194 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.064171    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclh8\" (UniqueName: \"kubernetes.io/projected/1f3bec10-78d2-4de0-803e-a844b37c72cb-kube-api-access-tclh8\") pod \"alertmanager-main-0\" (UID: \"1f3bec10-78d2-4de0-803e-a844b37c72cb\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.182449 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.182402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 14:41:41.333335 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.333232    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 14:41:41.336863 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:41:41.336827    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3bec10_78d2_4de0_803e_a844b37c72cb.slice/crio-480431e9fd9b756dc989e6068775cbf592dafa4bba39e7045c7cb2f79b55a3ea WatchSource:0}: Error finding container 480431e9fd9b756dc989e6068775cbf592dafa4bba39e7045c7cb2f79b55a3ea: Status 404 returned error can't find the container with id 480431e9fd9b756dc989e6068775cbf592dafa4bba39e7045c7cb2f79b55a3ea
Jul 01 14:41:41.789968 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.789939    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f3bec10-78d2-4de0-803e-a844b37c72cb" containerID="e7d11afd7fe0f150864e49802d5ee5e7e74273975ab2a07a1d7f4f80a0b7c174" exitCode=0
Jul 01 14:41:41.790170 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.790016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerDied","Data":"e7d11afd7fe0f150864e49802d5ee5e7e74273975ab2a07a1d7f4f80a0b7c174"}
Jul 01 14:41:41.790170 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.790084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"480431e9fd9b756dc989e6068775cbf592dafa4bba39e7045c7cb2f79b55a3ea"}
Jul 01 14:41:41.957524 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:41.957496    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e283ab02-2930-4626-9559-8aa25e3032e7" path="/var/lib/kubelet/pods/e283ab02-2930-4626-9559-8aa25e3032e7/volumes"
Jul 01 14:41:42.800697 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.800660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"693bd571d35e1cf3f5254ae603febbf9a569437fa77e44ba8f8f421417e8ec0a"}
Jul 01 14:41:42.801594 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.801570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"e8ceb0af72e3c0883b96583f9e3e61702d1ab4e813a95e22cdd06edc49bc2b83"}
Jul 01 14:41:42.801717 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.801705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"b3af018c5ac27f646929e8a37234ab96292a9db7d39f57b2210e0cbbac4343fe"}
Jul 01 14:41:42.801817 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.801806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"e7f773b09e00bb43a3f33f0302ab742f843db0a75e2e6f1d09acc33c29e27423"}
Jul 01 14:41:42.801933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.801919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"3e0491bc27814d6b44fe1b2d5984cffa42b086e09f7cea5fd5cb29164a131fd2"}
Jul 01 14:41:42.802071 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.802054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f3bec10-78d2-4de0-803e-a844b37c72cb","Type":"ContainerStarted","Data":"d7b8e8711d8734432372038d68606b2a422663ac0488806b505401467e393c76"}
Jul 01 14:41:42.837417 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:42.837360    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.819813655 podStartE2EDuration="2.837343108s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.529453ms" isStatefulPod=false podCreationTimestamp="2026-07-01 14:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:41:42.836500908 +0000 UTC m=+181.362784940" watchObservedRunningTime="2026-07-01 14:41:42.837343108 +0000 UTC m=+181.363627109"
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.277588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.278145    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="prometheus" containerID="cri-o://447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d" gracePeriod=600
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.278382    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-web" containerID="cri-o://a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0" gracePeriod=600
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.278410    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="config-reloader" containerID="cri-o://13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198" gracePeriod=600
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.278552    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b" gracePeriod=600
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.278556    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="thanos-sidecar" containerID="cri-o://53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2" gracePeriod=600
Jul 01 14:41:43.282591 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.278649    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy" containerID="cri-o://d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4" gracePeriod=600
Jul 01 14:41:43.538891 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.538827    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.677515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-1\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-tls-assets\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677727 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-db\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677727 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-web-config\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677727 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-metrics-client-ca\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677727 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677659    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677727 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.677727 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config-out\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.678017 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.677868    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.678071 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-tls\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.678124 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678059    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.678124 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-kube-rbac-proxy\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-serving-certs-ca-bundle\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-grpc-tls\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-metrics-client-certs\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x4wf\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-kube-api-access-9x4wf\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-2\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-trusted-ca-bundle\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-kubelet-serving-ca-bundle\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678944    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-thanos-prometheus-http-client-file\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678947    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.678976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-0\") pod \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\" (UID: \"54ca1efb-b7b0-43e6-8094-1f4dc17c104b\") "
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.679258    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.679277    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-metrics-client-ca\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.679291    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.680093 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.679586    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:41:43.680876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.680205    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config" (OuterVolumeSpecName: "config") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.680876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.680214    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config-out" (OuterVolumeSpecName: "config-out") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:41:43.680876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.680552    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.681264 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681237    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.681387 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681267    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.681387 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681332    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:41:43.681387 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681364    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.681559 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681404    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.681559 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681488    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:41:43.681835 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.681809    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.682124 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.682098    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.683060 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.683038    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-kube-api-access-9x4wf" (OuterVolumeSpecName: "kube-api-access-9x4wf") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "kube-api-access-9x4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:41:43.683267 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.683244    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.683430 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.683267    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.683430 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.683404    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.691496 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.691474    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-web-config" (OuterVolumeSpecName: "web-config") pod "54ca1efb-b7b0-43e6-8094-1f4dc17c104b" (UID: "54ca1efb-b7b0-43e6-8094-1f4dc17c104b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:41:43.780074 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780026    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780074 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780068    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780074 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780079    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780074 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780090    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780100    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780109    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-tls-assets\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780117    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-prometheus-k8s-db\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780147    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-web-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780155    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780162    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780170    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-config-out\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780178    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780187    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-kube-rbac-proxy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780194    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-grpc-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780202    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-metrics-client-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780211    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.780338 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.780220    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x4wf\" (UniqueName: \"kubernetes.io/projected/54ca1efb-b7b0-43e6-8094-1f4dc17c104b-kube-api-access-9x4wf\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:41:43.807247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807174    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b" exitCode=0
Jul 01 14:41:43.807247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807196    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4" exitCode=0
Jul 01 14:41:43.807247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807202    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0" exitCode=0
Jul 01 14:41:43.807247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807208    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2" exitCode=0
Jul 01 14:41:43.807247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807213    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198" exitCode=0
Jul 01 14:41:43.807247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807219    2579 generic.go:356] "Generic (PLEG): container finished" podID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d" exitCode=0
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807248    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"}
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"}
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"}
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"}
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807420    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"}
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"}
Jul 01 14:41:43.807787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.807476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"54ca1efb-b7b0-43e6-8094-1f4dc17c104b","Type":"ContainerDied","Data":"7d17ac94ea2843f0bb51aaa360f1748ecf959d5b7cdbfc05fd167c3475cf3cc1"}
Jul 01 14:41:43.816092 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.816057    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.822978 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.822952    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.829634 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.829616    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.835933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.835918    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.839667 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.839645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:41:43.843085 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.843069    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.847358 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.847339    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:41:43.851054 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.851035    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.859140 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.859119    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.859436 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.859416    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.859514 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.859443    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"} err="failed to get container status \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist"
Jul 01 14:41:43.859514 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.859461    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.859704 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.859689    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.859745 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.859711    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"} err="failed to get container status \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist"
Jul 01 14:41:43.859745 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.859727    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.860013 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.859991    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.860052 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860023    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"} err="failed to get container status \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist"
Jul 01 14:41:43.860052 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860045    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.860283 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.860265    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.860351 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860290    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"} err="failed to get container status \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist"
Jul 01 14:41:43.860351 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860322    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.860560 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.860544    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.860612 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860565    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"} err="failed to get container status \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist"
Jul 01 14:41:43.860612 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860578    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.860783 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.860768    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.860834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860786    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"} err="failed to get container status \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist"
Jul 01 14:41:43.860834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.860798    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.861054 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:41:43.861036    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.861097 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861059    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"} err="failed to get container status \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist"
Jul 01 14:41:43.861097 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861072    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.861272 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861256    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"} err="failed to get container status \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist"
Jul 01 14:41:43.861334 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861273    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.861496 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861479    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"} err="failed to get container status \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist"
Jul 01 14:41:43.861553 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861497    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.861700 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861684    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"} err="failed to get container status \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist"
Jul 01 14:41:43.861758 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861700    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.861923 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861908    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"} err="failed to get container status \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist"
Jul 01 14:41:43.861965 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.861923    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.862127 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862102    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"} err="failed to get container status \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist"
Jul 01 14:41:43.862199 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862129    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.862380 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862364    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"} err="failed to get container status \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist"
Jul 01 14:41:43.862430 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862379    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.862598 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862580    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"} err="failed to get container status \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist"
Jul 01 14:41:43.862657 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862600    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.862850 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862833    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"} err="failed to get container status \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist"
Jul 01 14:41:43.862917 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.862851    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.863061 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863041    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"} err="failed to get container status \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist"
Jul 01 14:41:43.863128 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863064    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.863320 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863286    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"} err="failed to get container status \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist"
Jul 01 14:41:43.863320 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863316    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.863512 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863494    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"} err="failed to get container status \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist"
Jul 01 14:41:43.863581 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863514    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.863719 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863701    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"} err="failed to get container status \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist"
Jul 01 14:41:43.863771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863720    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.863905 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863885    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"} err="failed to get container status \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist"
Jul 01 14:41:43.863949 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.863906    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.864137 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864115    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"} err="failed to get container status \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist"
Jul 01 14:41:43.864188 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864140    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.864391 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864375    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"} err="failed to get container status \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist"
Jul 01 14:41:43.864435 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864392    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.864582 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864563    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"} err="failed to get container status \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist"
Jul 01 14:41:43.864629 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864582    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.864793 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864769    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"} err="failed to get container status \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist"
Jul 01 14:41:43.864848 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.864796    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.865043 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865027    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"} err="failed to get container status \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist"
Jul 01 14:41:43.865084 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865043    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.865284 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865263    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"} err="failed to get container status \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist"
Jul 01 14:41:43.865337 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865286    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.865540 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865522    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"} err="failed to get container status \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist"
Jul 01 14:41:43.865589 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865541    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.865747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865726    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"} err="failed to get container status \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist"
Jul 01 14:41:43.865794 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865749    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.865941 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865919    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"} err="failed to get container status \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist"
Jul 01 14:41:43.865987 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.865941    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.866151 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"} err="failed to get container status \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist"
Jul 01 14:41:43.866210 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866152    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.866401 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866384    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"} err="failed to get container status \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist"
Jul 01 14:41:43.866447 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866402    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.866628 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866606    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"} err="failed to get container status \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist"
Jul 01 14:41:43.866712 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866630    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.866857 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866840    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"} err="failed to get container status \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist"
Jul 01 14:41:43.866904 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.866858    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.867197 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867163    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"} err="failed to get container status \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist"
Jul 01 14:41:43.867197 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867195    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.867544 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867524    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"} err="failed to get container status \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist"
Jul 01 14:41:43.867625 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867545    2579 scope.go:117] "RemoveContainer" containerID="7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"
Jul 01 14:41:43.867792 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867772    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b"} err="failed to get container status \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": rpc error: code = NotFound desc = could not find container \"7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b\": container with ID starting with 7a6809e2e8d5ead45b2f1f4589a9c5dc81b9600df69e89cecd0a6bd74de1b77b not found: ID does not exist"
Jul 01 14:41:43.867834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867794    2579 scope.go:117] "RemoveContainer" containerID="d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"
Jul 01 14:41:43.867993 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867966    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4"} err="failed to get container status \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": rpc error: code = NotFound desc = could not find container \"d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4\": container with ID starting with d66f7d8f78b50ee59d4388601d4f95c0979716c2601fe60034c6dc4c3873e8a4 not found: ID does not exist"
Jul 01 14:41:43.868080 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.867996    2579 scope.go:117] "RemoveContainer" containerID="a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"
Jul 01 14:41:43.868226 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868209    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0"} err="failed to get container status \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": rpc error: code = NotFound desc = could not find container \"a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0\": container with ID starting with a78003be381898d6d841faa96dc301266ed9e8ebeac8da2e3c150caf01cc9de0 not found: ID does not exist"
Jul 01 14:41:43.868295 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868227    2579 scope.go:117] "RemoveContainer" containerID="53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"
Jul 01 14:41:43.868486 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868464    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2"} err="failed to get container status \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": rpc error: code = NotFound desc = could not find container \"53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2\": container with ID starting with 53e92fa75c762e53c6f2a4d93d6d4050efc0fd4c3c95fe6211412b91bc614ad2 not found: ID does not exist"
Jul 01 14:41:43.868539 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868488    2579 scope.go:117] "RemoveContainer" containerID="13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"
Jul 01 14:41:43.868704 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868686    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198"} err="failed to get container status \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": rpc error: code = NotFound desc = could not find container \"13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198\": container with ID starting with 13fb7094b44d4be8aa1d19af3b0e262dc1642aa9992827cc4ff2a01abbd14198 not found: ID does not exist"
Jul 01 14:41:43.868743 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868706    2579 scope.go:117] "RemoveContainer" containerID="447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"
Jul 01 14:41:43.868934 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868916    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d"} err="failed to get container status \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": rpc error: code = NotFound desc = could not find container \"447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d\": container with ID starting with 447646e04051f98cf3cb3e6dce58c361105a0cf40f34a2d0695961945eef4d0d not found: ID does not exist"
Jul 01 14:41:43.869007 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.868934    2579 scope.go:117] "RemoveContainer" containerID="1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"
Jul 01 14:41:43.869145 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.869129    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7"} err="failed to get container status \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": rpc error: code = NotFound desc = could not find container \"1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7\": container with ID starting with 1fa15802482c4472a0aefa9b58eeec0b77fe21b979771c955a24e7aad22da4c7 not found: ID does not exist"
Jul 01 14:41:43.889279 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:41:43.889593 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889581    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy"
Jul 01 14:41:43.889632 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889595    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy"
Jul 01 14:41:43.889632 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889608    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-thanos"
Jul 01 14:41:43.889632 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889614    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-thanos"
Jul 01 14:41:43.889632 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="init-config-reloader"
Jul 01 14:41:43.889632 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="init-config-reloader"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889634    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="prometheus"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889640    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="prometheus"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889646    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="config-reloader"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889651    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="config-reloader"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="thanos-sidecar"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889661    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="thanos-sidecar"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889670    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-web"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-web"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889731    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="thanos-sidecar"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889738    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="prometheus"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889744    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="config-reloader"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889752    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-web"
Jul 01 14:41:43.889797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.889757    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" containerName="kube-rbac-proxy-thanos"
Jul 01 14:41:43.893450 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.893430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.898676 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.898654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 14:41:43.898837 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.898658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 14:41:43.898931 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.898908    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 14:41:43.898931 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.898918    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 14:41:43.899431 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899375    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9rnv4\""
Jul 01 14:41:43.899545 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 14:41:43.899545 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 14:41:43.899729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 14:41:43.899729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899620    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 14:41:43.899729 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-rou38k9vu717\""
Jul 01 14:41:43.899887 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 14:41:43.900047 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899905    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 14:41:43.900047 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.899941    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 14:41:43.900047 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.900019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 14:41:43.902652 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.902629    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 14:41:43.916966 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.916939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 14:41:43.936242 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.936213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:41:43.957006 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.956972    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ca1efb-b7b0-43e6-8094-1f4dc17c104b" path="/var/lib/kubelet/pods/54ca1efb-b7b0-43e6-8094-1f4dc17c104b/volumes"
Jul 01 14:41:43.982934 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.982897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.982934 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.982935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983170 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.982988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842t5\" (UniqueName: \"kubernetes.io/projected/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-kube-api-access-842t5\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983362 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-config\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983362 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983484 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983484 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983484 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983484 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983649 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983649 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983649 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983797 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983868 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983868 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983973 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:43.983973 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:43.983910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085407 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085407 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085407 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085407 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842t5\" (UniqueName: \"kubernetes.io/projected/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-kube-api-access-842t5\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-config\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.085851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.085927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086197    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086633 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.086831 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.086808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.087125 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.087103    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.088623 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.088595    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-config\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.088829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.088935    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.089538    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.089758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.089991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090769 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.090145    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090769 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.090235    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090769 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.090283    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090769 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.090541    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.090895 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.090877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.091117 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.091094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.091169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.091144    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.091294 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.091273    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.091746 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.091722    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.092192 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.092176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.095398 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.095378    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-842t5\" (UniqueName: \"kubernetes.io/projected/41bf6482-4ee4-42e3-bfab-a0d6c74959a5-kube-api-access-842t5\") pod \"prometheus-k8s-0\" (UID: \"41bf6482-4ee4-42e3-bfab-a0d6c74959a5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.203860 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.203824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:41:44.367978 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.367733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 14:41:44.370221 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:41:44.370195    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bf6482_4ee4_42e3_bfab_a0d6c74959a5.slice/crio-f9d1a75809fe182abd74a2623002427a4b9c14fae51285f5a5716a0049adf9cb WatchSource:0}: Error finding container f9d1a75809fe182abd74a2623002427a4b9c14fae51285f5a5716a0049adf9cb: Status 404 returned error can't find the container with id f9d1a75809fe182abd74a2623002427a4b9c14fae51285f5a5716a0049adf9cb
Jul 01 14:41:44.814490 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.814455    2579 generic.go:356] "Generic (PLEG): container finished" podID="41bf6482-4ee4-42e3-bfab-a0d6c74959a5" containerID="df1137dd8bf62cbfc1c7201dac906550ef3136043248b0d1fdfdbbef0faaa931" exitCode=0
Jul 01 14:41:44.814913 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.814505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerDied","Data":"df1137dd8bf62cbfc1c7201dac906550ef3136043248b0d1fdfdbbef0faaa931"}
Jul 01 14:41:44.814913 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:44.814593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"f9d1a75809fe182abd74a2623002427a4b9c14fae51285f5a5716a0049adf9cb"}
Jul 01 14:41:45.821801 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.821767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"e6fb9d5da23d0feee1eb050a66ee065101807db1685aff5057aa587c49151ba2"}
Jul 01 14:41:45.822206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.821834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"d576b38b9c41b10f627a6d0e5a5bf828aae7150598c9a3e92ca77db603a4618e"}
Jul 01 14:41:45.822206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.821868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"0f9dfaba7d638b86ee39bd0bc1999d47921a45f6effcaa3e8cf08b1a58ef1c4a"}
Jul 01 14:41:45.822206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.821895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"266084425b4bc589edbd7d23e4785408b499d65d2d498eabed8da20d741e9c9b"}
Jul 01 14:41:45.822206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.821919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"eb8b3b9ecfaeff7c3f787089e9c16071e1382f6501dba21c2ca3683f3b578c8b"}
Jul 01 14:41:45.822206 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.821950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bf6482-4ee4-42e3-bfab-a0d6c74959a5","Type":"ContainerStarted","Data":"11496d9c8870c2263d16eb85e1e597f045df44c616f72fe65c17607932564abb"}
Jul 01 14:41:45.853412 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:45.853359    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.834025366 podStartE2EDuration="2.853342603s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.317237ms" isStatefulPod=false podCreationTimestamp="2026-07-01 14:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:41:45.852725813 +0000 UTC m=+184.379009843" watchObservedRunningTime="2026-07-01 14:41:45.853342603 +0000 UTC m=+184.379626608"
Jul 01 14:41:49.204259 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:41:49.204214    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:42:44.205000 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:42:44.204958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:42:44.220379 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:42:44.220355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:42:45.084520 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:42:45.084490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 14:43:41.906567 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:43:41.906537    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:43:41.909787 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:43:41.909761    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 14:43:41.909971 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:43:41.909957    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:46:47.815763 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.815730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-224zh"]
Jul 01 14:46:47.819053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.819037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-224zh"
Jul 01 14:46:47.821590 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.821565    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 14:46:47.821709 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.821610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 14:46:47.822425 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.822404    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z7cqg\""
Jul 01 14:46:47.822520 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.822407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 14:46:47.827830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.827803    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-224zh"]
Jul 01 14:46:47.871146 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.871109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gwc\" (UniqueName: \"kubernetes.io/projected/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-kube-api-access-w9gwc\") pod \"s3-init-224zh\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") " pod="kserve/s3-init-224zh"
Jul 01 14:46:47.871351 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.871169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-model-cache\") pod \"s3-init-224zh\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") " pod="kserve/s3-init-224zh"
Jul 01 14:46:47.972134 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.972101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gwc\" (UniqueName: \"kubernetes.io/projected/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-kube-api-access-w9gwc\") pod \"s3-init-224zh\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") " pod="kserve/s3-init-224zh"
Jul 01 14:46:47.972320 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.972159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-model-cache\") pod \"s3-init-224zh\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") " pod="kserve/s3-init-224zh"
Jul 01 14:46:47.972536 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.972519    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-model-cache\") pod \"s3-init-224zh\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") " pod="kserve/s3-init-224zh"
Jul 01 14:46:47.982611 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:47.982585    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gwc\" (UniqueName: \"kubernetes.io/projected/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-kube-api-access-w9gwc\") pod \"s3-init-224zh\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") " pod="kserve/s3-init-224zh"
Jul 01 14:46:48.145438 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:48.145408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-224zh"
Jul 01 14:46:48.282028 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:48.281995    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-224zh"]
Jul 01 14:46:48.285069 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:46:48.285037    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d733f4_2e73_4994_9b20_49b0c9ab9dc2.slice/crio-1c1aaa1318f16c66a198e51fc1134f10a633a4f8e8d7a35769f2fc17a1d9a128 WatchSource:0}: Error finding container 1c1aaa1318f16c66a198e51fc1134f10a633a4f8e8d7a35769f2fc17a1d9a128: Status 404 returned error can't find the container with id 1c1aaa1318f16c66a198e51fc1134f10a633a4f8e8d7a35769f2fc17a1d9a128
Jul 01 14:46:48.286984 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:48.286968    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 14:46:49.067460 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:49.067419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-224zh" event={"ID":"61d733f4-2e73-4994-9b20-49b0c9ab9dc2","Type":"ContainerStarted","Data":"1c1aaa1318f16c66a198e51fc1134f10a633a4f8e8d7a35769f2fc17a1d9a128"}
Jul 01 14:46:52.085336 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:52.085279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-224zh" event={"ID":"61d733f4-2e73-4994-9b20-49b0c9ab9dc2","Type":"ContainerStarted","Data":"de670e156a5b28cd8f2a7d3843069355190b094ce43d8e5319bad4ea008a01d3"}
Jul 01 14:46:54.096136 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:54.096050    2579 generic.go:356] "Generic (PLEG): container finished" podID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerID="de670e156a5b28cd8f2a7d3843069355190b094ce43d8e5319bad4ea008a01d3" exitCode=0
Jul 01 14:46:54.096136 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:54.096124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-224zh" event={"ID":"61d733f4-2e73-4994-9b20-49b0c9ab9dc2","Type":"ContainerDied","Data":"de670e156a5b28cd8f2a7d3843069355190b094ce43d8e5319bad4ea008a01d3"}
Jul 01 14:46:59.123633 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:59.123545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-224zh" event={"ID":"61d733f4-2e73-4994-9b20-49b0c9ab9dc2","Type":"ContainerStarted","Data":"511b2cbea7845dc0e51372181e79388fbce9dbd1e38657ca6b58e630eeffc552"}
Jul 01 14:46:59.143144 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:46:59.143078    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-224zh" podStartSLOduration=2.203207783 podStartE2EDuration="12.143057924s" totalImagesPullingTime="8.282519639s" totalInitContainerRuntime="1.657330502s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:46:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:46:59.140843817 +0000 UTC m=+497.667127822" watchObservedRunningTime="2026-07-01 14:46:59.143057924 +0000 UTC m=+497.669341930"
Jul 01 14:48:04.389654 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:04.389621    2579 generic.go:356] "Generic (PLEG): container finished" podID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerID="511b2cbea7845dc0e51372181e79388fbce9dbd1e38657ca6b58e630eeffc552" exitCode=0
Jul 01 14:48:04.390143 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:04.389671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-224zh" event={"ID":"61d733f4-2e73-4994-9b20-49b0c9ab9dc2","Type":"ContainerDied","Data":"511b2cbea7845dc0e51372181e79388fbce9dbd1e38657ca6b58e630eeffc552"}
Jul 01 14:48:05.527770 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.527743    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-224zh"
Jul 01 14:48:05.651100 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.651002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-model-cache\") pod \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") "
Jul 01 14:48:05.651100 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.651061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9gwc\" (UniqueName: \"kubernetes.io/projected/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-kube-api-access-w9gwc\") pod \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\" (UID: \"61d733f4-2e73-4994-9b20-49b0c9ab9dc2\") "
Jul 01 14:48:05.653239 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.653200    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-kube-api-access-w9gwc" (OuterVolumeSpecName: "kube-api-access-w9gwc") pod "61d733f4-2e73-4994-9b20-49b0c9ab9dc2" (UID: "61d733f4-2e73-4994-9b20-49b0c9ab9dc2"). InnerVolumeSpecName "kube-api-access-w9gwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:48:05.668601 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.668576    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "61d733f4-2e73-4994-9b20-49b0c9ab9dc2" (UID: "61d733f4-2e73-4994-9b20-49b0c9ab9dc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:48:05.752120 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.752087    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:48:05.752120 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:05.752115    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gwc\" (UniqueName: \"kubernetes.io/projected/61d733f4-2e73-4994-9b20-49b0c9ab9dc2-kube-api-access-w9gwc\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:48:06.398772 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:06.398739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-224zh" event={"ID":"61d733f4-2e73-4994-9b20-49b0c9ab9dc2","Type":"ContainerDied","Data":"1c1aaa1318f16c66a198e51fc1134f10a633a4f8e8d7a35769f2fc17a1d9a128"}
Jul 01 14:48:06.398772 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:06.398777    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1aaa1318f16c66a198e51fc1134f10a633a4f8e8d7a35769f2fc17a1d9a128"
Jul 01 14:48:06.398968 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:06.398790    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-224zh"
Jul 01 14:48:13.656535 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.656500    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-7g9vd"]
Jul 01 14:48:13.657035 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.656835    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerName="download-hf-model"
Jul 01 14:48:13.657035 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.656846    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerName="download-hf-model"
Jul 01 14:48:13.657035 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.656870    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerName="s3-init"
Jul 01 14:48:13.657035 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.656875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerName="s3-init"
Jul 01 14:48:13.657035 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.656925    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d733f4-2e73-4994-9b20-49b0c9ab9dc2" containerName="s3-init"
Jul 01 14:48:13.660091 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.660075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.662423 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.662397    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 01 14:48:13.662578 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.662425    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 14:48:13.662578 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.662462    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z7cqg\""
Jul 01 14:48:13.663286 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.663271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 14:48:13.671755 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.671731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7g9vd"]
Jul 01 14:48:13.709784 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.709752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wt5\" (UniqueName: \"kubernetes.io/projected/ed8cd65b-5966-48f7-84a8-82391b9fae73-kube-api-access-s4wt5\") pod \"s3-tls-init-custom-7g9vd\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") " pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.709934 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.709815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd65b-5966-48f7-84a8-82391b9fae73-model-cache\") pod \"s3-tls-init-custom-7g9vd\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") " pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.810515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.810476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd65b-5966-48f7-84a8-82391b9fae73-model-cache\") pod \"s3-tls-init-custom-7g9vd\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") " pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.810684 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.810555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wt5\" (UniqueName: \"kubernetes.io/projected/ed8cd65b-5966-48f7-84a8-82391b9fae73-kube-api-access-s4wt5\") pod \"s3-tls-init-custom-7g9vd\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") " pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.810899 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.810881    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd65b-5966-48f7-84a8-82391b9fae73-model-cache\") pod \"s3-tls-init-custom-7g9vd\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") " pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.822352 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.822295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wt5\" (UniqueName: \"kubernetes.io/projected/ed8cd65b-5966-48f7-84a8-82391b9fae73-kube-api-access-s4wt5\") pod \"s3-tls-init-custom-7g9vd\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") " pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:13.981457 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:13.981369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:14.116279 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:14.116245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7g9vd"]
Jul 01 14:48:14.119700 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:48:14.119673    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded8cd65b_5966_48f7_84a8_82391b9fae73.slice/crio-13fc195c80ec4fdb4d3e39ed5179965a8a497e26df69bc3d0faf01cb8db337fb WatchSource:0}: Error finding container 13fc195c80ec4fdb4d3e39ed5179965a8a497e26df69bc3d0faf01cb8db337fb: Status 404 returned error can't find the container with id 13fc195c80ec4fdb4d3e39ed5179965a8a497e26df69bc3d0faf01cb8db337fb
Jul 01 14:48:14.432651 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:14.432607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7g9vd" event={"ID":"ed8cd65b-5966-48f7-84a8-82391b9fae73","Type":"ContainerStarted","Data":"f4f8fa8dbcf41004b618f6015df128eff28235f7f7ae3cff9f2a88922b8d0df5"}
Jul 01 14:48:14.432826 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:14.432663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7g9vd" event={"ID":"ed8cd65b-5966-48f7-84a8-82391b9fae73","Type":"ContainerStarted","Data":"13fc195c80ec4fdb4d3e39ed5179965a8a497e26df69bc3d0faf01cb8db337fb"}
Jul 01 14:48:16.442643 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:16.442607    2579 generic.go:356] "Generic (PLEG): container finished" podID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerID="f4f8fa8dbcf41004b618f6015df128eff28235f7f7ae3cff9f2a88922b8d0df5" exitCode=0
Jul 01 14:48:16.443048 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:16.442681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7g9vd" event={"ID":"ed8cd65b-5966-48f7-84a8-82391b9fae73","Type":"ContainerDied","Data":"f4f8fa8dbcf41004b618f6015df128eff28235f7f7ae3cff9f2a88922b8d0df5"}
Jul 01 14:48:17.448514 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:17.448480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7g9vd" event={"ID":"ed8cd65b-5966-48f7-84a8-82391b9fae73","Type":"ContainerStarted","Data":"ff434a6f6e18f1f648b39e7dadabbc06f17e66a2e244e785a35e0417c2dfd9a0"}
Jul 01 14:48:17.467846 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:17.467795    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-7g9vd" podStartSLOduration=2.416742938 podStartE2EDuration="4.467779421s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.051036483s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:48:17.465877075 +0000 UTC m=+575.992161078" watchObservedRunningTime="2026-07-01 14:48:17.467779421 +0000 UTC m=+575.994063423"
Jul 01 14:48:23.483282 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:23.483248    2579 generic.go:356] "Generic (PLEG): container finished" podID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerID="ff434a6f6e18f1f648b39e7dadabbc06f17e66a2e244e785a35e0417c2dfd9a0" exitCode=0
Jul 01 14:48:23.483681 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:23.483328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7g9vd" event={"ID":"ed8cd65b-5966-48f7-84a8-82391b9fae73","Type":"ContainerDied","Data":"ff434a6f6e18f1f648b39e7dadabbc06f17e66a2e244e785a35e0417c2dfd9a0"}
Jul 01 14:48:24.638298 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.638275    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:24.702459 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.702424    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4wt5\" (UniqueName: \"kubernetes.io/projected/ed8cd65b-5966-48f7-84a8-82391b9fae73-kube-api-access-s4wt5\") pod \"ed8cd65b-5966-48f7-84a8-82391b9fae73\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") "
Jul 01 14:48:24.702459 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.702460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd65b-5966-48f7-84a8-82391b9fae73-model-cache\") pod \"ed8cd65b-5966-48f7-84a8-82391b9fae73\" (UID: \"ed8cd65b-5966-48f7-84a8-82391b9fae73\") "
Jul 01 14:48:24.704513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.704483    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8cd65b-5966-48f7-84a8-82391b9fae73-kube-api-access-s4wt5" (OuterVolumeSpecName: "kube-api-access-s4wt5") pod "ed8cd65b-5966-48f7-84a8-82391b9fae73" (UID: "ed8cd65b-5966-48f7-84a8-82391b9fae73"). InnerVolumeSpecName "kube-api-access-s4wt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:48:24.724319 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.724271    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8cd65b-5966-48f7-84a8-82391b9fae73-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed8cd65b-5966-48f7-84a8-82391b9fae73" (UID: "ed8cd65b-5966-48f7-84a8-82391b9fae73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:48:24.803088 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.803006    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4wt5\" (UniqueName: \"kubernetes.io/projected/ed8cd65b-5966-48f7-84a8-82391b9fae73-kube-api-access-s4wt5\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:48:24.803088 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:24.803033    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8cd65b-5966-48f7-84a8-82391b9fae73-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:48:25.492581 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:25.492552    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7g9vd"
Jul 01 14:48:25.492761 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:25.492587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7g9vd" event={"ID":"ed8cd65b-5966-48f7-84a8-82391b9fae73","Type":"ContainerDied","Data":"13fc195c80ec4fdb4d3e39ed5179965a8a497e26df69bc3d0faf01cb8db337fb"}
Jul 01 14:48:25.492761 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:25.492627    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13fc195c80ec4fdb4d3e39ed5179965a8a497e26df69bc3d0faf01cb8db337fb"
Jul 01 14:48:28.891275 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.891241    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-fd7xr"]
Jul 01 14:48:28.891767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.891605    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerName="download-hf-model"
Jul 01 14:48:28.891767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.891618    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerName="download-hf-model"
Jul 01 14:48:28.891767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.891627    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerName="s3-tls-init-custom"
Jul 01 14:48:28.891767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.891632    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerName="s3-tls-init-custom"
Jul 01 14:48:28.891767 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.891687    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed8cd65b-5966-48f7-84a8-82391b9fae73" containerName="s3-tls-init-custom"
Jul 01 14:48:28.894838 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.894821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:28.897455 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.897435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 01 14:48:28.897602 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.897435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 14:48:28.898136 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.898122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z7cqg\""
Jul 01 14:48:28.898179 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.898158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 14:48:28.903348 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.903325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-fd7xr"]
Jul 01 14:48:28.938218 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.938183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199f6c7-cdf9-46c4-9136-c64797996d94-model-cache\") pod \"s3-tls-init-serving-fd7xr\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") " pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:28.938432 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:28.938229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkt5\" (UniqueName: \"kubernetes.io/projected/d199f6c7-cdf9-46c4-9136-c64797996d94-kube-api-access-vpkt5\") pod \"s3-tls-init-serving-fd7xr\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") " pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:29.039256 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.039214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199f6c7-cdf9-46c4-9136-c64797996d94-model-cache\") pod \"s3-tls-init-serving-fd7xr\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") " pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:29.039468 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.039270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkt5\" (UniqueName: \"kubernetes.io/projected/d199f6c7-cdf9-46c4-9136-c64797996d94-kube-api-access-vpkt5\") pod \"s3-tls-init-serving-fd7xr\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") " pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:29.039762 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.039741    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199f6c7-cdf9-46c4-9136-c64797996d94-model-cache\") pod \"s3-tls-init-serving-fd7xr\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") " pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:29.049687 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.049660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkt5\" (UniqueName: \"kubernetes.io/projected/d199f6c7-cdf9-46c4-9136-c64797996d94-kube-api-access-vpkt5\") pod \"s3-tls-init-serving-fd7xr\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") " pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:29.212143 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.212062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:29.347707 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.347677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-fd7xr"]
Jul 01 14:48:29.350096 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:48:29.350070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd199f6c7_cdf9_46c4_9136_c64797996d94.slice/crio-387590628f75834ea78d27b7fe696b783b3d0cfe59884c140d099806883f6b0f WatchSource:0}: Error finding container 387590628f75834ea78d27b7fe696b783b3d0cfe59884c140d099806883f6b0f: Status 404 returned error can't find the container with id 387590628f75834ea78d27b7fe696b783b3d0cfe59884c140d099806883f6b0f
Jul 01 14:48:29.512585 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.512498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fd7xr" event={"ID":"d199f6c7-cdf9-46c4-9136-c64797996d94","Type":"ContainerStarted","Data":"236b83f349cb6c078d401e4f7bb747bc713b4403b08455a0d179a18abc341b39"}
Jul 01 14:48:29.512585 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:29.512541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fd7xr" event={"ID":"d199f6c7-cdf9-46c4-9136-c64797996d94","Type":"ContainerStarted","Data":"387590628f75834ea78d27b7fe696b783b3d0cfe59884c140d099806883f6b0f"}
Jul 01 14:48:31.527169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:31.527082    2579 generic.go:356] "Generic (PLEG): container finished" podID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerID="236b83f349cb6c078d401e4f7bb747bc713b4403b08455a0d179a18abc341b39" exitCode=0
Jul 01 14:48:31.527169 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:31.527152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fd7xr" event={"ID":"d199f6c7-cdf9-46c4-9136-c64797996d94","Type":"ContainerDied","Data":"236b83f349cb6c078d401e4f7bb747bc713b4403b08455a0d179a18abc341b39"}
Jul 01 14:48:32.532954 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:32.532920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fd7xr" event={"ID":"d199f6c7-cdf9-46c4-9136-c64797996d94","Type":"ContainerStarted","Data":"b72c06160005b2dfe067739e80686670ea0b35a8f62e4b2755971f6a50211549"}
Jul 01 14:48:32.552145 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:32.552095    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-fd7xr" podStartSLOduration=2.783400322 podStartE2EDuration="4.552079525s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.768679203s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:48:32.551518959 +0000 UTC m=+591.077802964" watchObservedRunningTime="2026-07-01 14:48:32.552079525 +0000 UTC m=+591.078363528"
Jul 01 14:48:37.559633 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:37.559597    2579 generic.go:356] "Generic (PLEG): container finished" podID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerID="b72c06160005b2dfe067739e80686670ea0b35a8f62e4b2755971f6a50211549" exitCode=0
Jul 01 14:48:37.560005 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:37.559666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fd7xr" event={"ID":"d199f6c7-cdf9-46c4-9136-c64797996d94","Type":"ContainerDied","Data":"b72c06160005b2dfe067739e80686670ea0b35a8f62e4b2755971f6a50211549"}
Jul 01 14:48:38.704556 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.704529    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:38.831902 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.831804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199f6c7-cdf9-46c4-9136-c64797996d94-model-cache\") pod \"d199f6c7-cdf9-46c4-9136-c64797996d94\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") "
Jul 01 14:48:38.831902 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.831877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpkt5\" (UniqueName: \"kubernetes.io/projected/d199f6c7-cdf9-46c4-9136-c64797996d94-kube-api-access-vpkt5\") pod \"d199f6c7-cdf9-46c4-9136-c64797996d94\" (UID: \"d199f6c7-cdf9-46c4-9136-c64797996d94\") "
Jul 01 14:48:38.833909 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.833875    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d199f6c7-cdf9-46c4-9136-c64797996d94-kube-api-access-vpkt5" (OuterVolumeSpecName: "kube-api-access-vpkt5") pod "d199f6c7-cdf9-46c4-9136-c64797996d94" (UID: "d199f6c7-cdf9-46c4-9136-c64797996d94"). InnerVolumeSpecName "kube-api-access-vpkt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:48:38.855730 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.855688    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d199f6c7-cdf9-46c4-9136-c64797996d94-model-cache" (OuterVolumeSpecName: "model-cache") pod "d199f6c7-cdf9-46c4-9136-c64797996d94" (UID: "d199f6c7-cdf9-46c4-9136-c64797996d94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:48:38.932889 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.932846    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d199f6c7-cdf9-46c4-9136-c64797996d94-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:48:38.932889 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:38.932890    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkt5\" (UniqueName: \"kubernetes.io/projected/d199f6c7-cdf9-46c4-9136-c64797996d94-kube-api-access-vpkt5\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:48:39.571235 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:39.571204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fd7xr"
Jul 01 14:48:39.571235 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:39.571226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fd7xr" event={"ID":"d199f6c7-cdf9-46c4-9136-c64797996d94","Type":"ContainerDied","Data":"387590628f75834ea78d27b7fe696b783b3d0cfe59884c140d099806883f6b0f"}
Jul 01 14:48:39.571526 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:39.571257    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387590628f75834ea78d27b7fe696b783b3d0cfe59884c140d099806883f6b0f"
Jul 01 14:48:41.937854 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:41.937826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:48:41.939369 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:41.939346    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:48:52.088252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.088218    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"]
Jul 01 14:48:52.088672 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.088570    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerName="download-hf-model"
Jul 01 14:48:52.088672 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.088584    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerName="download-hf-model"
Jul 01 14:48:52.088672 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.088591    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerName="s3-tls-init-serving"
Jul 01 14:48:52.088672 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.088597    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerName="s3-tls-init-serving"
Jul 01 14:48:52.088672 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.088666    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d199f6c7-cdf9-46c4-9136-c64797996d94" containerName="s3-tls-init-serving"
Jul 01 14:48:52.093189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.093172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.095872 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.095850    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4cvh9\""
Jul 01 14:48:52.095995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.095906    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:48:52.095995 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.095906    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 14:48:52.096321 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.096293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 14:48:52.096426 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.096411    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 01 14:48:52.096651 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.096635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 01 14:48:52.096740 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.096662    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 14:48:52.108272 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.108245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"]
Jul 01 14:48:52.251329 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.251266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxd74\" (UniqueName: \"kubernetes.io/projected/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-api-access-dxd74\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.251515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.251340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6bc5bd8-e432-4809-96d2-f34f761665ca-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.251515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.251377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.251515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.251412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6bc5bd8-e432-4809-96d2-f34f761665ca-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.251515 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.251457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.352285 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.352189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxd74\" (UniqueName: \"kubernetes.io/projected/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-api-access-dxd74\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.352285 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.352234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6bc5bd8-e432-4809-96d2-f34f761665ca-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.352513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.352373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.352513 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.352446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6bc5bd8-e432-4809-96d2-f34f761665ca-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.352617 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.352511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.352905 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.352874    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6bc5bd8-e432-4809-96d2-f34f761665ca-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.353137 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.353118    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.353193 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.353126    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.354742 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.354717    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6bc5bd8-e432-4809-96d2-f34f761665ca-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.362053 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.362031    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxd74\" (UniqueName: \"kubernetes.io/projected/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-api-access-dxd74\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.406694 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.406657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:48:52.547229 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.547203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"]
Jul 01 14:48:52.549808 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:48:52.549779    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bc5bd8_e432_4809_96d2_f34f761665ca.slice/crio-d35fa0b4efca898ac50a7f280fca818823deb34736591036c53bfbc143d6f2b1 WatchSource:0}: Error finding container d35fa0b4efca898ac50a7f280fca818823deb34736591036c53bfbc143d6f2b1: Status 404 returned error can't find the container with id d35fa0b4efca898ac50a7f280fca818823deb34736591036c53bfbc143d6f2b1
Jul 01 14:48:52.634806 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.634715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerStarted","Data":"dc358d1b1e9778f40799ea0e1538d7013506b0f8159072a789875ece7bee7199"}
Jul 01 14:48:52.634806 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:52.634790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerStarted","Data":"d35fa0b4efca898ac50a7f280fca818823deb34736591036c53bfbc143d6f2b1"}
Jul 01 14:48:56.655242 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:56.655210    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerID="dc358d1b1e9778f40799ea0e1538d7013506b0f8159072a789875ece7bee7199" exitCode=0
Jul 01 14:48:56.655659 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:48:56.655269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerDied","Data":"dc358d1b1e9778f40799ea0e1538d7013506b0f8159072a789875ece7bee7199"}
Jul 01 14:49:11.746681 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:11.746633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerStarted","Data":"c16fcd0a43ae177d80f4860ae066bf489193b18e59b4345e83db4879da379437"}
Jul 01 14:49:13.760068 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:13.760031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerStarted","Data":"b6f787df9d12ae4cbc86c44784fe91493d2ea2cbf141659665db2e5b4b724a82"}
Jul 01 14:49:16.777211 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:16.777169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerStarted","Data":"0a1a26462c049646cc0b6e5b5808537eee600dcf457654ffa76dd29429607a48"}
Jul 01 14:49:16.777697 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:16.777404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:49:16.777697 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:16.777466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:49:16.778699 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:16.778670    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:16.805265 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:16.805213    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podStartSLOduration=1.404747452 podStartE2EDuration="24.805195747s" totalImagesPullingTime="19.84513212s" totalInitContainerRuntime="3.555316175s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:48:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:49:16.802286825 +0000 UTC m=+635.328570830" watchObservedRunningTime="2026-07-01 14:49:16.805195747 +0000 UTC m=+635.331479752"
Jul 01 14:49:17.782230 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:17.782187    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:49:17.782673 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:17.782394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:17.783447 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:17.783421    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:18.786861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:18.786818    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:18.787376 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:18.787224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:18.790734 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:18.790714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:49:19.790981 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:19.790935    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:19.791450 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:19.791202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:29.791609 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:29.791559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:29.792111 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:29.792029    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:39.791151 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:39.791103    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:39.791711 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:39.791579    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:49.791642 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:49.791595    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:49.792168 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:49.792055    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:49:59.791179 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:59.791131    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:49:59.791680 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:49:59.791573    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:09.791488 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:09.791441    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:50:09.791977 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:09.791921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:19.791158 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:19.791105    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:50:19.791608 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:19.791556    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:29.792441 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:29.792410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:50:29.792848 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:29.792519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:50:37.218573 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.218494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"]
Jul 01 14:50:37.218983 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.218748    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" containerID="cri-o://c16fcd0a43ae177d80f4860ae066bf489193b18e59b4345e83db4879da379437" gracePeriod=30
Jul 01 14:50:37.218983 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.218801    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" containerID="cri-o://0a1a26462c049646cc0b6e5b5808537eee600dcf457654ffa76dd29429607a48" gracePeriod=30
Jul 01 14:50:37.218983 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.218805    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" containerID="cri-o://b6f787df9d12ae4cbc86c44784fe91493d2ea2cbf141659665db2e5b4b724a82" gracePeriod=30
Jul 01 14:50:37.233126 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.233103    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"]
Jul 01 14:50:37.235991 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.235975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.238393 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.238373    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 01 14:50:37.238499 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.238470    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 01 14:50:37.255011 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.254984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"]
Jul 01 14:50:37.363186 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.363151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1209b15f-ff14-452d-8c5c-c873bf868e57-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.363370 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.363202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.363370 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.363266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.363370 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.363352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.363504 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.363405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq26l\" (UniqueName: \"kubernetes.io/projected/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-api-access-gq26l\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.464696 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.464656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq26l\" (UniqueName: \"kubernetes.io/projected/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-api-access-gq26l\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.464898 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.464809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1209b15f-ff14-452d-8c5c-c873bf868e57-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.464898 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.464848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.464898 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.464885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.465076 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.464924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.465076 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:50:37.465061    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 01 14:50:37.465171 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:50:37.465118    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls podName:1209b15f-ff14-452d-8c5c-c873bf868e57 nodeName:}" failed. No retries permitted until 2026-07-01 14:50:37.965099107 +0000 UTC m=+716.491383089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" (UID: "1209b15f-ff14-452d-8c5c-c873bf868e57") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 01 14:50:37.465289 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.465269    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1209b15f-ff14-452d-8c5c-c873bf868e57-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.465701 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.465680    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.465740 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.465689    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.476533 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.476471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq26l\" (UniqueName: \"kubernetes.io/projected/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-api-access-gq26l\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.968415 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.968383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:37.970783 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:37.970754    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:38.149251 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:38.149218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:38.152907 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:38.152884    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerID="b6f787df9d12ae4cbc86c44784fe91493d2ea2cbf141659665db2e5b4b724a82" exitCode=2
Jul 01 14:50:38.153021 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:38.152958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerDied","Data":"b6f787df9d12ae4cbc86c44784fe91493d2ea2cbf141659665db2e5b4b724a82"}
Jul 01 14:50:38.285237 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:38.285212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"]
Jul 01 14:50:38.287669 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:50:38.287636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1209b15f_ff14_452d_8c5c_c873bf868e57.slice/crio-36609ab04654e304b44506318e3e55c4c6b1a7f5f8c0078495cadcc65cd7814e WatchSource:0}: Error finding container 36609ab04654e304b44506318e3e55c4c6b1a7f5f8c0078495cadcc65cd7814e: Status 404 returned error can't find the container with id 36609ab04654e304b44506318e3e55c4c6b1a7f5f8c0078495cadcc65cd7814e
Jul 01 14:50:38.787719 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:38.787677    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 14:50:39.159293 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:39.159247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerStarted","Data":"1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d"}
Jul 01 14:50:39.159490 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:39.159346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerStarted","Data":"36609ab04654e304b44506318e3e55c4c6b1a7f5f8c0078495cadcc65cd7814e"}
Jul 01 14:50:39.791189 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:39.791146    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:50:39.791643 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:39.791475    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:42.176315 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:42.176279    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerID="c16fcd0a43ae177d80f4860ae066bf489193b18e59b4345e83db4879da379437" exitCode=0
Jul 01 14:50:42.176764 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:42.176358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerDied","Data":"c16fcd0a43ae177d80f4860ae066bf489193b18e59b4345e83db4879da379437"}
Jul 01 14:50:42.177781 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:42.177762    2579 generic.go:356] "Generic (PLEG): container finished" podID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerID="1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d" exitCode=0
Jul 01 14:50:42.177893 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:42.177796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerDied","Data":"1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d"}
Jul 01 14:50:43.184861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:43.184828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerStarted","Data":"a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9"}
Jul 01 14:50:43.185264 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:43.184895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerStarted","Data":"fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3"}
Jul 01 14:50:43.185264 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:43.184924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerStarted","Data":"b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5"}
Jul 01 14:50:43.185264 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:43.185147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:43.211916 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:43.211861    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podStartSLOduration=2.596838972 podStartE2EDuration="6.211845217s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.615006245s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:50:43.209912352 +0000 UTC m=+721.736196380" watchObservedRunningTime="2026-07-01 14:50:43.211845217 +0000 UTC m=+721.738129220"
Jul 01 14:50:43.787881 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:43.787842    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 14:50:44.189852 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:44.189820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:44.190227 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:44.189899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:44.191360 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:44.191330    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:50:44.191963 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:44.191938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:45.194647 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:45.194606    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:50:45.195114 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:45.195089    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:48.787944 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:48.787892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 14:50:48.788383 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:48.788041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:50:49.791141 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:49.791092    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:50:49.791596 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:49.791409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:50.199918 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:50.199886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:50:50.200494 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:50.200451    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:50:50.200823 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:50.200803    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:53.787386 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:53.787336    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 14:50:58.787785 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:58.787737    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 14:50:59.791832 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:59.791794    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 01 14:50:59.792252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:59.791956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:50:59.792252 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:59.792161    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:50:59.792351 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:50:59.792270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:51:00.201314 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:00.201273    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:51:00.201823 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:00.201792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:51:03.787039 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:03.786993    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 01 14:51:07.314114 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.314082    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerID="0a1a26462c049646cc0b6e5b5808537eee600dcf457654ffa76dd29429607a48" exitCode=0
Jul 01 14:51:07.314458 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.314160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerDied","Data":"0a1a26462c049646cc0b6e5b5808537eee600dcf457654ffa76dd29429607a48"}
Jul 01 14:51:07.368266 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.368239    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:51:07.423179 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxd74\" (UniqueName: \"kubernetes.io/projected/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-api-access-dxd74\") pod \"b6bc5bd8-e432-4809-96d2-f34f761665ca\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") "
Jul 01 14:51:07.423394 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6bc5bd8-e432-4809-96d2-f34f761665ca-proxy-tls\") pod \"b6bc5bd8-e432-4809-96d2-f34f761665ca\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") "
Jul 01 14:51:07.423394 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423216    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-cabundle-cert\") pod \"b6bc5bd8-e432-4809-96d2-f34f761665ca\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") "
Jul 01 14:51:07.423394 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-rbac-proxy-sar-config\") pod \"b6bc5bd8-e432-4809-96d2-f34f761665ca\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") "
Jul 01 14:51:07.423394 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6bc5bd8-e432-4809-96d2-f34f761665ca-kserve-provision-location\") pod \"b6bc5bd8-e432-4809-96d2-f34f761665ca\" (UID: \"b6bc5bd8-e432-4809-96d2-f34f761665ca\") "
Jul 01 14:51:07.423703 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423667    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6bc5bd8-e432-4809-96d2-f34f761665ca" (UID: "b6bc5bd8-e432-4809-96d2-f34f761665ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:51:07.423829 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423768    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bc5bd8-e432-4809-96d2-f34f761665ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6bc5bd8-e432-4809-96d2-f34f761665ca" (UID: "b6bc5bd8-e432-4809-96d2-f34f761665ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:51:07.423829 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.423771    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6bc5bd8-e432-4809-96d2-f34f761665ca" (UID: "b6bc5bd8-e432-4809-96d2-f34f761665ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:51:07.425560 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.425529    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bc5bd8-e432-4809-96d2-f34f761665ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6bc5bd8-e432-4809-96d2-f34f761665ca" (UID: "b6bc5bd8-e432-4809-96d2-f34f761665ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:51:07.425658 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.425575    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-api-access-dxd74" (OuterVolumeSpecName: "kube-api-access-dxd74") pod "b6bc5bd8-e432-4809-96d2-f34f761665ca" (UID: "b6bc5bd8-e432-4809-96d2-f34f761665ca"). InnerVolumeSpecName "kube-api-access-dxd74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:51:07.525089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.525006    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6bc5bd8-e432-4809-96d2-f34f761665ca-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:51:07.525089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.525032    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxd74\" (UniqueName: \"kubernetes.io/projected/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-api-access-dxd74\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:51:07.525089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.525045    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6bc5bd8-e432-4809-96d2-f34f761665ca-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:51:07.525089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.525056    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:51:07.525089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:07.525065    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6bc5bd8-e432-4809-96d2-f34f761665ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:51:08.321809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.321726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg" event={"ID":"b6bc5bd8-e432-4809-96d2-f34f761665ca","Type":"ContainerDied","Data":"d35fa0b4efca898ac50a7f280fca818823deb34736591036c53bfbc143d6f2b1"}
Jul 01 14:51:08.321809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.321752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"
Jul 01 14:51:08.321809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.321799    2579 scope.go:117] "RemoveContainer" containerID="0a1a26462c049646cc0b6e5b5808537eee600dcf457654ffa76dd29429607a48"
Jul 01 14:51:08.330473 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.330451    2579 scope.go:117] "RemoveContainer" containerID="b6f787df9d12ae4cbc86c44784fe91493d2ea2cbf141659665db2e5b4b724a82"
Jul 01 14:51:08.338451 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.338430    2579 scope.go:117] "RemoveContainer" containerID="c16fcd0a43ae177d80f4860ae066bf489193b18e59b4345e83db4879da379437"
Jul 01 14:51:08.343615 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.343594    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"]
Jul 01 14:51:08.346145 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.346127    2579 scope.go:117] "RemoveContainer" containerID="dc358d1b1e9778f40799ea0e1538d7013506b0f8159072a789875ece7bee7199"
Jul 01 14:51:08.351299 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:08.351274    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-wxjhg"]
Jul 01 14:51:09.957224 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:09.957189    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" path="/var/lib/kubelet/pods/b6bc5bd8-e432-4809-96d2-f34f761665ca/volumes"
Jul 01 14:51:10.200696 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:10.200655    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:51:10.201131 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:10.201108    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:51:20.200887 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:20.200840    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:51:20.201416 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:20.201338    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:51:30.200658 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:30.200608    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:51:30.201236 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:30.201075    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:51:40.200648 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:40.200601    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:51:40.201081 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:40.201052    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:51:50.200452 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:50.200402    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:51:50.200945 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:51:50.200902    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:00.201565 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:00.201529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:00.201997 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:00.201696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:12.436939 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:12.436899    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"]
Jul 01 14:52:12.437837 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:12.437203    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" containerID="cri-o://b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5" gracePeriod=30
Jul 01 14:52:12.437837 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:12.437247    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" containerID="cri-o://fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3" gracePeriod=30
Jul 01 14:52:12.437837 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:12.437475    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" containerID="cri-o://a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9" gracePeriod=30
Jul 01 14:52:12.621540 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:12.621508    2579 generic.go:356] "Generic (PLEG): container finished" podID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerID="fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3" exitCode=2
Jul 01 14:52:12.621705 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:12.621583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerDied","Data":"fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3"}
Jul 01 14:52:15.195009 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:15.194965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 14:52:17.647288 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:17.647253    2579 generic.go:356] "Generic (PLEG): container finished" podID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerID="b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5" exitCode=0
Jul 01 14:52:17.647684 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:17.647333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerDied","Data":"b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5"}
Jul 01 14:52:20.195445 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:20.195406    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 14:52:20.200866 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:20.200838    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:52:20.201170 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:20.201134    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:25.194882 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:25.194831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 14:52:25.195331 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:25.194968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:30.195555 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:30.195513    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 14:52:30.200933 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:30.200894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:52:30.201284 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:30.201258    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:32.472739 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.472699    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"]
Jul 01 14:52:32.473126 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473099    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="storage-initializer"
Jul 01 14:52:32.473126 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473112    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="storage-initializer"
Jul 01 14:52:32.473198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473130    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent"
Jul 01 14:52:32.473198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent"
Jul 01 14:52:32.473198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473149    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy"
Jul 01 14:52:32.473198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473155    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy"
Jul 01 14:52:32.473198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473163    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container"
Jul 01 14:52:32.473198 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473169    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container"
Jul 01 14:52:32.473403 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473221    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kserve-container"
Jul 01 14:52:32.473403 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473230    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="agent"
Jul 01 14:52:32.473403 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.473256    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bc5bd8-e432-4809-96d2-f34f761665ca" containerName="kube-rbac-proxy"
Jul 01 14:52:32.477013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.476995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.479314 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.479276    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 01 14:52:32.479448 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.479430    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 01 14:52:32.489366 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.489344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"]
Jul 01 14:52:32.662019 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.661975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr44f\" (UniqueName: \"kubernetes.io/projected/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-api-access-zr44f\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.662207 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.662091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.662207 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.662153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.662207 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.662173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1259e9e8-2423-4505-a7c1-9386b81bd74f-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.662355 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.662223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1259e9e8-2423-4505-a7c1-9386b81bd74f-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763587 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763587 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr44f\" (UniqueName: \"kubernetes.io/projected/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-api-access-zr44f\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763587 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:52:32.763486    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 01 14:52:32.763587 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:52:32.763582    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls podName:1259e9e8-2423-4505-a7c1-9386b81bd74f nodeName:}" failed. No retries permitted until 2026-07-01 14:52:33.263556983 +0000 UTC m=+831.789840974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls") pod "isvc-logger-predictor-599b89d7c-gr9rm" (UID: "1259e9e8-2423-4505-a7c1-9386b81bd74f") : secret "isvc-logger-predictor-serving-cert" not found
Jul 01 14:52:32.763834 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763773    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1259e9e8-2423-4505-a7c1-9386b81bd74f-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.763937 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763918    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.764012 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.763995    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:32.772823 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:32.772798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr44f\" (UniqueName: \"kubernetes.io/projected/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-api-access-zr44f\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:33.267762 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:33.267722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:33.270164 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:33.270145    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-gr9rm\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:33.386784 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:33.386745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:33.534221 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:33.534199    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"]
Jul 01 14:52:33.536956 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:52:33.536927    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1259e9e8_2423_4505_a7c1_9386b81bd74f.slice/crio-17f5a4eedc92b041a6e1f6c4ca53367c5d349fa076f72455c4503fc71ca4c114 WatchSource:0}: Error finding container 17f5a4eedc92b041a6e1f6c4ca53367c5d349fa076f72455c4503fc71ca4c114: Status 404 returned error can't find the container with id 17f5a4eedc92b041a6e1f6c4ca53367c5d349fa076f72455c4503fc71ca4c114
Jul 01 14:52:33.730684 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:33.730650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerStarted","Data":"1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273"}
Jul 01 14:52:33.730861 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:33.730714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerStarted","Data":"17f5a4eedc92b041a6e1f6c4ca53367c5d349fa076f72455c4503fc71ca4c114"}
Jul 01 14:52:35.195452 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:35.195405    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 14:52:37.750054 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:37.749955    2579 generic.go:356] "Generic (PLEG): container finished" podID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerID="1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273" exitCode=0
Jul 01 14:52:37.750054 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:37.750007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerDied","Data":"1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273"}
Jul 01 14:52:38.756723 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.756685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerStarted","Data":"d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8"}
Jul 01 14:52:38.757172 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.756756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerStarted","Data":"7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99"}
Jul 01 14:52:38.757172 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.756781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerStarted","Data":"467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85"}
Jul 01 14:52:38.757172 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.757087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:38.757172 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.757157    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:38.758573 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.758546    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:52:38.784074 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:38.784018    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podStartSLOduration=2.945568497 podStartE2EDuration="6.784003744s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.838435247s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:52:38.782451476 +0000 UTC m=+837.308735480" watchObservedRunningTime="2026-07-01 14:52:38.784003744 +0000 UTC m=+837.310287748"
Jul 01 14:52:39.762250 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:39.762217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:39.762734 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:39.762206    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:52:39.763375 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:39.763344    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:40.195634 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.195591    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 01 14:52:40.201090 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.201056    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 01 14:52:40.201197 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.201182    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:40.201444 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.201421    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:40.201551 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.201537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:40.766645 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.766597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:52:40.767128 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:40.767078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:42.590258 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.590225    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:42.641892 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.641860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-rbac-proxy-sar-config\") pod \"1209b15f-ff14-452d-8c5c-c873bf868e57\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") "
Jul 01 14:52:42.642086 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.641900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1209b15f-ff14-452d-8c5c-c873bf868e57-kserve-provision-location\") pod \"1209b15f-ff14-452d-8c5c-c873bf868e57\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") "
Jul 01 14:52:42.642086 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.641940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq26l\" (UniqueName: \"kubernetes.io/projected/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-api-access-gq26l\") pod \"1209b15f-ff14-452d-8c5c-c873bf868e57\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") "
Jul 01 14:52:42.642086 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.641978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-cabundle-cert\") pod \"1209b15f-ff14-452d-8c5c-c873bf868e57\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") "
Jul 01 14:52:42.642298 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.642261    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1209b15f-ff14-452d-8c5c-c873bf868e57" (UID: "1209b15f-ff14-452d-8c5c-c873bf868e57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:52:42.642444 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.642332    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1209b15f-ff14-452d-8c5c-c873bf868e57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1209b15f-ff14-452d-8c5c-c873bf868e57" (UID: "1209b15f-ff14-452d-8c5c-c873bf868e57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:52:42.642514 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.642463    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1209b15f-ff14-452d-8c5c-c873bf868e57" (UID: "1209b15f-ff14-452d-8c5c-c873bf868e57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:52:42.644079 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.644062    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-api-access-gq26l" (OuterVolumeSpecName: "kube-api-access-gq26l") pod "1209b15f-ff14-452d-8c5c-c873bf868e57" (UID: "1209b15f-ff14-452d-8c5c-c873bf868e57"). InnerVolumeSpecName "kube-api-access-gq26l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:52:42.742659 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.742575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls\") pod \"1209b15f-ff14-452d-8c5c-c873bf868e57\" (UID: \"1209b15f-ff14-452d-8c5c-c873bf868e57\") "
Jul 01 14:52:42.742810 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.742790    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:52:42.742810 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.742802    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:52:42.742890 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.742810    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1209b15f-ff14-452d-8c5c-c873bf868e57-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:52:42.742890 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.742819    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq26l\" (UniqueName: \"kubernetes.io/projected/1209b15f-ff14-452d-8c5c-c873bf868e57-kube-api-access-gq26l\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:52:42.744564 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.744538    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1209b15f-ff14-452d-8c5c-c873bf868e57" (UID: "1209b15f-ff14-452d-8c5c-c873bf868e57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:52:42.776992 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.776948    2579 generic.go:356] "Generic (PLEG): container finished" podID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerID="a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9" exitCode=0
Jul 01 14:52:42.777149 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.776987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerDied","Data":"a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9"}
Jul 01 14:52:42.777149 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.777017    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"
Jul 01 14:52:42.777149 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.777088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf" event={"ID":"1209b15f-ff14-452d-8c5c-c873bf868e57","Type":"ContainerDied","Data":"36609ab04654e304b44506318e3e55c4c6b1a7f5f8c0078495cadcc65cd7814e"}
Jul 01 14:52:42.777149 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.777121    2579 scope.go:117] "RemoveContainer" containerID="a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9"
Jul 01 14:52:42.785358 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.785340    2579 scope.go:117] "RemoveContainer" containerID="fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3"
Jul 01 14:52:42.793205 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.793174    2579 scope.go:117] "RemoveContainer" containerID="b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5"
Jul 01 14:52:42.801365 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.801343    2579 scope.go:117] "RemoveContainer" containerID="1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d"
Jul 01 14:52:42.808294 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.808268    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"]
Jul 01 14:52:42.809502 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.809484    2579 scope.go:117] "RemoveContainer" containerID="a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9"
Jul 01 14:52:42.809807 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:52:42.809784    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9\": container with ID starting with a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9 not found: ID does not exist" containerID="a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9"
Jul 01 14:52:42.809860 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.809813    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9"} err="failed to get container status \"a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9\": rpc error: code = NotFound desc = could not find container \"a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9\": container with ID starting with a89e792fc5910226032463bd10a14766f31771f95263f7978154a4b4f26436b9 not found: ID does not exist"
Jul 01 14:52:42.809860 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.809833    2579 scope.go:117] "RemoveContainer" containerID="fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3"
Jul 01 14:52:42.810076 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:52:42.810058    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3\": container with ID starting with fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3 not found: ID does not exist" containerID="fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3"
Jul 01 14:52:42.810123 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.810083    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3"} err="failed to get container status \"fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3\": rpc error: code = NotFound desc = could not find container \"fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3\": container with ID starting with fa05d7fb1dfc6f10740d79c5c5006b1bd26061b7d14a55dbfb29cc2481834ff3 not found: ID does not exist"
Jul 01 14:52:42.810123 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.810099    2579 scope.go:117] "RemoveContainer" containerID="b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5"
Jul 01 14:52:42.810330 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:52:42.810289    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5\": container with ID starting with b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5 not found: ID does not exist" containerID="b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5"
Jul 01 14:52:42.810438 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.810329    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5"} err="failed to get container status \"b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5\": rpc error: code = NotFound desc = could not find container \"b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5\": container with ID starting with b2e20b3b0b8cc9b379a28ea69e013585ac62d7bd698ac7ee4c6bc51dcc79d6c5 not found: ID does not exist"
Jul 01 14:52:42.810438 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.810345    2579 scope.go:117] "RemoveContainer" containerID="1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d"
Jul 01 14:52:42.810593 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:52:42.810577    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d\": container with ID starting with 1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d not found: ID does not exist" containerID="1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d"
Jul 01 14:52:42.810634 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.810599    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d"} err="failed to get container status \"1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d\": rpc error: code = NotFound desc = could not find container \"1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d\": container with ID starting with 1600fd11f7505214a9ddeb14e23f442ce379a201aa387ec27b432afaf56bed1d not found: ID does not exist"
Jul 01 14:52:42.813566 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.813548    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-zhssf"]
Jul 01 14:52:42.843404 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:42.843369    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1209b15f-ff14-452d-8c5c-c873bf868e57-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:52:43.957232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:43.957195    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" path="/var/lib/kubelet/pods/1209b15f-ff14-452d-8c5c-c873bf868e57/volumes"
Jul 01 14:52:45.771374 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:45.771342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:52:45.772067 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:45.772037    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:52:45.772318 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:45.772285    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:52:55.772466 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:55.772412    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:52:55.772939 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:52:55.772832    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:05.772588 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:05.772544    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:53:05.773085 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:05.772894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:15.772033 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:15.771992    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:53:15.772590 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:15.772394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:25.772792 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:25.772747    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:53:25.773340 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:25.773236    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:35.772254 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:35.772156    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:53:35.772918 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:35.772875    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:41.969602 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:41.969574    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:53:41.970011 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:41.969670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:53:45.772851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:45.772803    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:53:45.773382 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:45.773218    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:53:50.952978 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:50.952923    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:53:50.953422 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:53:50.953297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:00.953965 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:00.953925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:00.954394 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:00.954012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:07.956914 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.956863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"]
Jul 01 14:54:07.957433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.957104    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" containerID="cri-o://467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85" gracePeriod=30
Jul 01 14:54:07.957433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.957123    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" containerID="cri-o://d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8" gracePeriod=30
Jul 01 14:54:07.957433 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.957193    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" containerID="cri-o://7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99" gracePeriod=30
Jul 01 14:54:07.991247 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991214    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"]
Jul 01 14:54:07.991641 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991625    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy"
Jul 01 14:54:07.991641 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991642    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy"
Jul 01 14:54:07.991747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991651    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="storage-initializer"
Jul 01 14:54:07.991747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991658    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="storage-initializer"
Jul 01 14:54:07.991747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991669    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container"
Jul 01 14:54:07.991747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991679    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container"
Jul 01 14:54:07.991747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991699    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent"
Jul 01 14:54:07.991747 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991704    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent"
Jul 01 14:54:07.991931 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991763    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="agent"
Jul 01 14:54:07.991931 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991773    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kserve-container"
Jul 01 14:54:07.991931 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.991779    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1209b15f-ff14-452d-8c5c-c873bf868e57" containerName="kube-rbac-proxy"
Jul 01 14:54:07.996649 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:07.996629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.000232 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.000212    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 01 14:54:08.000851 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.000830    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 01 14:54:08.025777 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.025750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"]
Jul 01 14:54:08.072705 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.072678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.072876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.072714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.072876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.072777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5g42\" (UniqueName: \"kubernetes.io/projected/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-api-access-f5g42\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.072876 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.072817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.073033 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.072890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.173756 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.173673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.173756 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.173710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.173756 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.173757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5g42\" (UniqueName: \"kubernetes.io/projected/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-api-access-f5g42\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.174039 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.173785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.174039 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.174010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.174185 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.174159    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.174695 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.174671    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.174809 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.174773    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.176335 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.176292    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.180245 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.180222    2579 generic.go:356] "Generic (PLEG): container finished" podID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerID="7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99" exitCode=2
Jul 01 14:54:08.180357 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.180254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerDied","Data":"7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99"}
Jul 01 14:54:08.185663 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.185645    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5g42\" (UniqueName: \"kubernetes.io/projected/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-api-access-f5g42\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.306915 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.306811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:54:08.444171 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:08.444139    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"]
Jul 01 14:54:08.447664 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:54:08.447635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4fdd56_7fa4_43c1_96f7_af5cae9d0572.slice/crio-682ea69e1cbd8376b7a8d1c673d14860fdd6674e957095acc477eec6cac4949a WatchSource:0}: Error finding container 682ea69e1cbd8376b7a8d1c673d14860fdd6674e957095acc477eec6cac4949a: Status 404 returned error can't find the container with id 682ea69e1cbd8376b7a8d1c673d14860fdd6674e957095acc477eec6cac4949a
Jul 01 14:54:09.191887 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:09.191849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerStarted","Data":"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"}
Jul 01 14:54:09.192405 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:09.191942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerStarted","Data":"682ea69e1cbd8376b7a8d1c673d14860fdd6674e957095acc477eec6cac4949a"}
Jul 01 14:54:10.766888 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:10.766841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 14:54:10.953072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:10.953024    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:54:10.953445 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:10.953412    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:13.212142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:13.212108    2579 generic.go:356] "Generic (PLEG): container finished" podID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerID="467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85" exitCode=0
Jul 01 14:54:13.212546 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:13.212183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerDied","Data":"467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85"}
Jul 01 14:54:15.767134 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:15.767094    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 14:54:20.767772 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:20.767731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 14:54:20.768228 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:20.767891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:20.953005 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:20.952958    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:54:20.953342 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:20.953319    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:25.767319 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:25.767271    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 14:54:30.767762 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:30.767714    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 14:54:30.953736 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:30.953688    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 01 14:54:30.953931 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:30.953894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:30.954090 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:30.954061    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 14:54:30.954188 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:30.954176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:35.767101 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:35.767057    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 01 14:54:38.144316 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.144286    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:38.216459 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls\") pod \"1259e9e8-2423-4505-a7c1-9386b81bd74f\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") "
Jul 01 14:54:38.216459 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-rbac-proxy-sar-config\") pod \"1259e9e8-2423-4505-a7c1-9386b81bd74f\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") "
Jul 01 14:54:38.216459 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-cabundle-cert\") pod \"1259e9e8-2423-4505-a7c1-9386b81bd74f\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") "
Jul 01 14:54:38.216745 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1259e9e8-2423-4505-a7c1-9386b81bd74f-kserve-provision-location\") pod \"1259e9e8-2423-4505-a7c1-9386b81bd74f\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") "
Jul 01 14:54:38.216745 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr44f\" (UniqueName: \"kubernetes.io/projected/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-api-access-zr44f\") pod \"1259e9e8-2423-4505-a7c1-9386b81bd74f\" (UID: \"1259e9e8-2423-4505-a7c1-9386b81bd74f\") "
Jul 01 14:54:38.216857 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216829    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1259e9e8-2423-4505-a7c1-9386b81bd74f" (UID: "1259e9e8-2423-4505-a7c1-9386b81bd74f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:54:38.216917 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216841    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1259e9e8-2423-4505-a7c1-9386b81bd74f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1259e9e8-2423-4505-a7c1-9386b81bd74f" (UID: "1259e9e8-2423-4505-a7c1-9386b81bd74f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:54:38.216960 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.216926    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1259e9e8-2423-4505-a7c1-9386b81bd74f" (UID: "1259e9e8-2423-4505-a7c1-9386b81bd74f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:54:38.218642 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.218615    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-api-access-zr44f" (OuterVolumeSpecName: "kube-api-access-zr44f") pod "1259e9e8-2423-4505-a7c1-9386b81bd74f" (UID: "1259e9e8-2423-4505-a7c1-9386b81bd74f"). InnerVolumeSpecName "kube-api-access-zr44f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:54:38.218943 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.218930    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1259e9e8-2423-4505-a7c1-9386b81bd74f" (UID: "1259e9e8-2423-4505-a7c1-9386b81bd74f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:54:38.317201 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.317160    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1259e9e8-2423-4505-a7c1-9386b81bd74f-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:54:38.317201 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.317195    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr44f\" (UniqueName: \"kubernetes.io/projected/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-api-access-zr44f\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:54:38.317201 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.317205    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1259e9e8-2423-4505-a7c1-9386b81bd74f-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:54:38.317201 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.317215    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:54:38.317503 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.317224    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1259e9e8-2423-4505-a7c1-9386b81bd74f-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:54:38.337282 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.337247    2579 generic.go:356] "Generic (PLEG): container finished" podID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerID="d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8" exitCode=137
Jul 01 14:54:38.337424 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.337337    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"
Jul 01 14:54:38.337424 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.337339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerDied","Data":"d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8"}
Jul 01 14:54:38.337424 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.337416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm" event={"ID":"1259e9e8-2423-4505-a7c1-9386b81bd74f","Type":"ContainerDied","Data":"17f5a4eedc92b041a6e1f6c4ca53367c5d349fa076f72455c4503fc71ca4c114"}
Jul 01 14:54:38.337542 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.337453    2579 scope.go:117] "RemoveContainer" containerID="d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8"
Jul 01 14:54:38.346209 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.346191    2579 scope.go:117] "RemoveContainer" containerID="7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99"
Jul 01 14:54:38.353744 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.353728    2579 scope.go:117] "RemoveContainer" containerID="467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85"
Jul 01 14:54:38.361078 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.361052    2579 scope.go:117] "RemoveContainer" containerID="1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273"
Jul 01 14:54:38.367752 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.367731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"]
Jul 01 14:54:38.370455 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.370432    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-gr9rm"]
Jul 01 14:54:38.370635 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.370623    2579 scope.go:117] "RemoveContainer" containerID="d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8"
Jul 01 14:54:38.370859 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:54:38.370843    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8\": container with ID starting with d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8 not found: ID does not exist" containerID="d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8"
Jul 01 14:54:38.370908 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.370867    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8"} err="failed to get container status \"d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8\": rpc error: code = NotFound desc = could not find container \"d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8\": container with ID starting with d9790baf7da431e28003b02aa283628c028b1502746d2d0c5278c214a1964ba8 not found: ID does not exist"
Jul 01 14:54:38.370908 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.370886    2579 scope.go:117] "RemoveContainer" containerID="7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99"
Jul 01 14:54:38.371122 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:54:38.371108    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99\": container with ID starting with 7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99 not found: ID does not exist" containerID="7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99"
Jul 01 14:54:38.371165 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.371125    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99"} err="failed to get container status \"7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99\": rpc error: code = NotFound desc = could not find container \"7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99\": container with ID starting with 7f1c1129e8273e51000bc7f19588f6b3ad00c9a4a6aff35d70f1bc4b16b0db99 not found: ID does not exist"
Jul 01 14:54:38.371165 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.371137    2579 scope.go:117] "RemoveContainer" containerID="467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85"
Jul 01 14:54:38.371381 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:54:38.371364    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85\": container with ID starting with 467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85 not found: ID does not exist" containerID="467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85"
Jul 01 14:54:38.371427 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.371389    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85"} err="failed to get container status \"467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85\": rpc error: code = NotFound desc = could not find container \"467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85\": container with ID starting with 467aec1f2f9df033a2a74ce9f21f01608322ae9e61ff778e9873b3c35f437a85 not found: ID does not exist"
Jul 01 14:54:38.371427 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.371405    2579 scope.go:117] "RemoveContainer" containerID="1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273"
Jul 01 14:54:38.371617 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:54:38.371602    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273\": container with ID starting with 1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273 not found: ID does not exist" containerID="1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273"
Jul 01 14:54:38.371655 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:38.371622    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273"} err="failed to get container status \"1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273\": rpc error: code = NotFound desc = could not find container \"1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273\": container with ID starting with 1ebd2c7c8aa4a87d3bd51b65e3c84e8e8b4295758e774a3fe850275b93755273 not found: ID does not exist"
Jul 01 14:54:39.343989 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:39.343905    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerID="fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238" exitCode=0
Jul 01 14:54:39.343989 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:39.343974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerDied","Data":"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"}
Jul 01 14:54:39.345470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:39.345454    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 14:54:39.960013 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:54:39.959976    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" path="/var/lib/kubelet/pods/1259e9e8-2423-4505-a7c1-9386b81bd74f/volumes"
Jul 01 14:56:43.994078 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:43.994041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerStarted","Data":"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"}
Jul 01 14:56:43.994538 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:43.994105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerStarted","Data":"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"}
Jul 01 14:56:43.994538 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:43.994240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:56:43.994538 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:43.994396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:56:43.995728 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:43.995701    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 14:56:44.998849 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:44.998803    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 14:56:50.003160 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:50.003128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:56:50.003762 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:50.003736    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 01 14:56:50.027771 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:56:50.027704    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" podStartSLOduration=8.268670734 podStartE2EDuration="2m43.027685615s" totalImagesPullingTime="2m4.308388817s" totalInitContainerRuntime="30.450626064s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:56:44.027908741 +0000 UTC m=+1082.554192749" watchObservedRunningTime="2026-07-01 14:56:50.027685615 +0000 UTC m=+1088.553969618"
Jul 01 14:57:00.005174 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:00.005142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:57:07.974333 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:07.974277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"]
Jul 01 14:57:07.974820 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:07.974650    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container" containerID="cri-o://50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab" gracePeriod=30
Jul 01 14:57:07.974820 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:07.974661    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kube-rbac-proxy" containerID="cri-o://081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782" gracePeriod=30
Jul 01 14:57:08.951021 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:08.950989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:57:09.078109 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-proxy-tls\") pod \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") "
Jul 01 14:57:09.078109 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078111    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5g42\" (UniqueName: \"kubernetes.io/projected/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-api-access-f5g42\") pod \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") "
Jul 01 14:57:09.078644 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078138    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-rbac-proxy-sar-config\") pod \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") "
Jul 01 14:57:09.078644 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078172    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-cabundle-cert\") pod \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") "
Jul 01 14:57:09.078644 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kserve-provision-location\") pod \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\" (UID: \"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572\") "
Jul 01 14:57:09.078644 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078555    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" (UID: "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:57:09.078644 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.078614    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" (UID: "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:57:09.080374 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.080348    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" (UID: "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:57:09.080479 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.080431    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-api-access-f5g42" (OuterVolumeSpecName: "kube-api-access-f5g42") pod "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" (UID: "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572"). InnerVolumeSpecName "kube-api-access-f5g42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:57:09.081141 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.081119    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" (UID: "5e4fdd56-7fa4-43c1-96f7-af5cae9d0572"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:57:09.122877 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122793    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerID="081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782" exitCode=2
Jul 01 14:57:09.122877 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122822    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerID="50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab" exitCode=0
Jul 01 14:57:09.122877 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122842    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"
Jul 01 14:57:09.123089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerDied","Data":"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"}
Jul 01 14:57:09.123089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerDied","Data":"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"}
Jul 01 14:57:09.123089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp" event={"ID":"5e4fdd56-7fa4-43c1-96f7-af5cae9d0572","Type":"ContainerDied","Data":"682ea69e1cbd8376b7a8d1c673d14860fdd6674e957095acc477eec6cac4949a"}
Jul 01 14:57:09.123089 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.122979    2579 scope.go:117] "RemoveContainer" containerID="081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"
Jul 01 14:57:09.132165 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.132149    2579 scope.go:117] "RemoveContainer" containerID="50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"
Jul 01 14:57:09.139778 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.139759    2579 scope.go:117] "RemoveContainer" containerID="fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"
Jul 01 14:57:09.147504 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.147484    2579 scope.go:117] "RemoveContainer" containerID="081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"
Jul 01 14:57:09.147887 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:57:09.147867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782\": container with ID starting with 081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782 not found: ID does not exist" containerID="081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"
Jul 01 14:57:09.147961 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.147893    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"} err="failed to get container status \"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782\": rpc error: code = NotFound desc = could not find container \"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782\": container with ID starting with 081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782 not found: ID does not exist"
Jul 01 14:57:09.147961 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.147911    2579 scope.go:117] "RemoveContainer" containerID="50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"
Jul 01 14:57:09.148202 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:57:09.148183    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab\": container with ID starting with 50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab not found: ID does not exist" containerID="50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"
Jul 01 14:57:09.148248 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.148212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"} err="failed to get container status \"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab\": rpc error: code = NotFound desc = could not find container \"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab\": container with ID starting with 50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab not found: ID does not exist"
Jul 01 14:57:09.148248 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.148229    2579 scope.go:117] "RemoveContainer" containerID="fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"
Jul 01 14:57:09.148625 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:57:09.148603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238\": container with ID starting with fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238 not found: ID does not exist" containerID="fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"
Jul 01 14:57:09.148673 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.148633    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"} err="failed to get container status \"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238\": rpc error: code = NotFound desc = could not find container \"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238\": container with ID starting with fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238 not found: ID does not exist"
Jul 01 14:57:09.148673 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.148654    2579 scope.go:117] "RemoveContainer" containerID="081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"
Jul 01 14:57:09.148925 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.148905    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782"} err="failed to get container status \"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782\": rpc error: code = NotFound desc = could not find container \"081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782\": container with ID starting with 081f4bad3dd1e37a12eefd7547032dfdd6f63c2ae722f0371a3c4ac561a3b782 not found: ID does not exist"
Jul 01 14:57:09.149010 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.148927    2579 scope.go:117] "RemoveContainer" containerID="50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"
Jul 01 14:57:09.149170 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.149149    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab"} err="failed to get container status \"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab\": rpc error: code = NotFound desc = could not find container \"50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab\": container with ID starting with 50e943b027c05b868b9b1ab554a6bfa1a4186fcd4daa9beda0cd1cd4848f59ab not found: ID does not exist"
Jul 01 14:57:09.149219 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.149171    2579 scope.go:117] "RemoveContainer" containerID="fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"
Jul 01 14:57:09.149424 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.149405    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238"} err="failed to get container status \"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238\": rpc error: code = NotFound desc = could not find container \"fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238\": container with ID starting with fa76feb4b9a6b7c47f90089a597794ce99164ac02f28d5cef66af214c6047238 not found: ID does not exist"
Jul 01 14:57:09.149559 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.149543    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"]
Jul 01 14:57:09.154383 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.154362    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-d8hqp"]
Jul 01 14:57:09.179249 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.179217    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:57:09.179249 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.179245    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5g42\" (UniqueName: \"kubernetes.io/projected/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-api-access-f5g42\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:57:09.179249 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.179256    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:57:09.179479 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.179266    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:57:09.179479 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.179277    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:57:09.957164 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:09.957130    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" path="/var/lib/kubelet/pods/5e4fdd56-7fa4-43c1-96f7-af5cae9d0572/volumes"
Jul 01 14:57:13.394919 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.394881    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"]
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395259    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395272    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395293    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="storage-initializer"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395299    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="storage-initializer"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395318    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395324    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kube-rbac-proxy"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395340    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kube-rbac-proxy"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395359    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395375    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="storage-initializer"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395380    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="storage-initializer"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395386    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container"
Jul 01 14:57:13.395413 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395390    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container"
Jul 01 14:57:13.395841 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395443    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kserve-container"
Jul 01 14:57:13.395841 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395452    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="kube-rbac-proxy"
Jul 01 14:57:13.395841 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395461    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1259e9e8-2423-4505-a7c1-9386b81bd74f" containerName="agent"
Jul 01 14:57:13.395841 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395466    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kserve-container"
Jul 01 14:57:13.395841 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.395472    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4fdd56-7fa4-43c1-96f7-af5cae9d0572" containerName="kube-rbac-proxy"
Jul 01 14:57:13.419132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.419098    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"]
Jul 01 14:57:13.419344 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.419244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.424416 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.424391    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 14:57:13.424572 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.424439    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 14:57:13.424572 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.424392    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 14:57:13.425006 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.424983    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 01 14:57:13.425150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.425008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:57:13.425150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.425036    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4cvh9\""
Jul 01 14:57:13.425150 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.425145    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 01 14:57:13.515031 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.514992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.515031 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.515038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.515374 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.515142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b88f574-0d17-4647-b5ab-26a607f24c9c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.515374 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.515198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5rt\" (UniqueName: \"kubernetes.io/projected/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-api-access-pz5rt\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.515374 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.515232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b88f574-0d17-4647-b5ab-26a607f24c9c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b88f574-0d17-4647-b5ab-26a607f24c9c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616342 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5rt\" (UniqueName: \"kubernetes.io/projected/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-api-access-pz5rt\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616342 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b88f574-0d17-4647-b5ab-26a607f24c9c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616342 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616510 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616720 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616698    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b88f574-0d17-4647-b5ab-26a607f24c9c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.616917 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.616901    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.617039 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.617021    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.618508 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.618489    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b88f574-0d17-4647-b5ab-26a607f24c9c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.628030 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.628005    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5rt\" (UniqueName: \"kubernetes.io/projected/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-api-access-pz5rt\") pod \"isvc-autogluon-v2-predictor-67b849d456-665mj\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.730242 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.730149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:13.908282 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:13.908244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"]
Jul 01 14:57:13.914072 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:57:13.914012    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b88f574_0d17_4647_b5ab_26a607f24c9c.slice/crio-09e96d4b0ee37b3ddfbda021014f638d57a34d3bf9bb29e222b5be9b1215c6ae WatchSource:0}: Error finding container 09e96d4b0ee37b3ddfbda021014f638d57a34d3bf9bb29e222b5be9b1215c6ae: Status 404 returned error can't find the container with id 09e96d4b0ee37b3ddfbda021014f638d57a34d3bf9bb29e222b5be9b1215c6ae
Jul 01 14:57:14.150061 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:14.150017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerStarted","Data":"7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770"}
Jul 01 14:57:14.150233 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:14.150076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerStarted","Data":"09e96d4b0ee37b3ddfbda021014f638d57a34d3bf9bb29e222b5be9b1215c6ae"}
Jul 01 14:57:43.286978 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:43.286939    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerID="7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770" exitCode=0
Jul 01 14:57:43.287431 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:43.287016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerDied","Data":"7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770"}
Jul 01 14:57:44.293638 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:44.293603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerStarted","Data":"502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999"}
Jul 01 14:57:44.294132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:44.293663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerStarted","Data":"4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9"}
Jul 01 14:57:44.294132 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:44.293907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:44.320165 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:44.320104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" podStartSLOduration=2.900620596 podStartE2EDuration="31.320089256s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.41946866s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:57:44.319400199 +0000 UTC m=+1142.845684216" watchObservedRunningTime="2026-07-01 14:57:44.320089256 +0000 UTC m=+1142.846373259"
Jul 01 14:57:45.298872 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:45.298842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:57:51.309207 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:57:51.309180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:58:42.002034 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:58:42.002007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:58:42.002639 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:58:42.002003    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 14:59:21.959878 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:21.959840    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:59:23.545800 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:23.545764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"]
Jul 01 14:59:23.546262 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:23.546031    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kserve-container" containerID="cri-o://4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9" gracePeriod=30
Jul 01 14:59:23.546262 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:23.546078    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kube-rbac-proxy" containerID="cri-o://502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999" gracePeriod=30
Jul 01 14:59:23.757289 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:23.757251    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerID="502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999" exitCode=2
Jul 01 14:59:23.757489 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:23.757336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerDied","Data":"502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999"}
Jul 01 14:59:24.492922 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.492882    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:59:24.660362 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b88f574-0d17-4647-b5ab-26a607f24c9c-kserve-provision-location\") pod \"3b88f574-0d17-4647-b5ab-26a607f24c9c\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") "
Jul 01 14:59:24.660830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-rbac-proxy-sar-config\") pod \"3b88f574-0d17-4647-b5ab-26a607f24c9c\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") "
Jul 01 14:59:24.660830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b88f574-0d17-4647-b5ab-26a607f24c9c-proxy-tls\") pod \"3b88f574-0d17-4647-b5ab-26a607f24c9c\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") "
Jul 01 14:59:24.660830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-cabundle-cert\") pod \"3b88f574-0d17-4647-b5ab-26a607f24c9c\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") "
Jul 01 14:59:24.660830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660511    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz5rt\" (UniqueName: \"kubernetes.io/projected/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-api-access-pz5rt\") pod \"3b88f574-0d17-4647-b5ab-26a607f24c9c\" (UID: \"3b88f574-0d17-4647-b5ab-26a607f24c9c\") "
Jul 01 14:59:24.660830 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660795    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b88f574-0d17-4647-b5ab-26a607f24c9c" (UID: "3b88f574-0d17-4647-b5ab-26a607f24c9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:24.661121 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.660917    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b88f574-0d17-4647-b5ab-26a607f24c9c" (UID: "3b88f574-0d17-4647-b5ab-26a607f24c9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 14:59:24.662619 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.662587    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-api-access-pz5rt" (OuterVolumeSpecName: "kube-api-access-pz5rt") pod "3b88f574-0d17-4647-b5ab-26a607f24c9c" (UID: "3b88f574-0d17-4647-b5ab-26a607f24c9c"). InnerVolumeSpecName "kube-api-access-pz5rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 14:59:24.662737 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.662614    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b88f574-0d17-4647-b5ab-26a607f24c9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b88f574-0d17-4647-b5ab-26a607f24c9c" (UID: "3b88f574-0d17-4647-b5ab-26a607f24c9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 14:59:24.665166 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.665140    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b88f574-0d17-4647-b5ab-26a607f24c9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b88f574-0d17-4647-b5ab-26a607f24c9c" (UID: "3b88f574-0d17-4647-b5ab-26a607f24c9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 14:59:24.761180 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.761151    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:59:24.761180 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.761177    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b88f574-0d17-4647-b5ab-26a607f24c9c-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:59:24.761180 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.761188    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b88f574-0d17-4647-b5ab-26a607f24c9c-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:59:24.761480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.761198    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5rt\" (UniqueName: \"kubernetes.io/projected/3b88f574-0d17-4647-b5ab-26a607f24c9c-kube-api-access-pz5rt\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:59:24.761480 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.761206    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b88f574-0d17-4647-b5ab-26a607f24c9c-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 14:59:24.764001 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.763970    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerID="4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9" exitCode=0
Jul 01 14:59:24.764072 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.764036    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"
Jul 01 14:59:24.764126 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.764063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerDied","Data":"4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9"}
Jul 01 14:59:24.764188 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.764145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj" event={"ID":"3b88f574-0d17-4647-b5ab-26a607f24c9c","Type":"ContainerDied","Data":"09e96d4b0ee37b3ddfbda021014f638d57a34d3bf9bb29e222b5be9b1215c6ae"}
Jul 01 14:59:24.764241 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.764192    2579 scope.go:117] "RemoveContainer" containerID="502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999"
Jul 01 14:59:24.773371 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.773355    2579 scope.go:117] "RemoveContainer" containerID="4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9"
Jul 01 14:59:24.781384 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.781363    2579 scope.go:117] "RemoveContainer" containerID="7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770"
Jul 01 14:59:24.788837 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.788820    2579 scope.go:117] "RemoveContainer" containerID="502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999"
Jul 01 14:59:24.789109 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:59:24.789090    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999\": container with ID starting with 502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999 not found: ID does not exist" containerID="502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999"
Jul 01 14:59:24.789154 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.789119    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999"} err="failed to get container status \"502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999\": rpc error: code = NotFound desc = could not find container \"502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999\": container with ID starting with 502771b670ac404888560e89d1a3f19e49f8daa2f585fc4cc4b1d85ab054b999 not found: ID does not exist"
Jul 01 14:59:24.789154 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.789137    2579 scope.go:117] "RemoveContainer" containerID="4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9"
Jul 01 14:59:24.789416 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:59:24.789393    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9\": container with ID starting with 4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9 not found: ID does not exist" containerID="4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9"
Jul 01 14:59:24.789528 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.789421    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9"} err="failed to get container status \"4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9\": rpc error: code = NotFound desc = could not find container \"4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9\": container with ID starting with 4670439c101a1caa2baa2b0fc818db37a14eac081b3d4cbd262dad214241c4b9 not found: ID does not exist"
Jul 01 14:59:24.789528 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.789446    2579 scope.go:117] "RemoveContainer" containerID="7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770"
Jul 01 14:59:24.789793 ip-10-0-128-59 kubenswrapper[2579]: E0701 14:59:24.789767    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770\": container with ID starting with 7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770 not found: ID does not exist" containerID="7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770"
Jul 01 14:59:24.789907 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.789797    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770"} err="failed to get container status \"7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770\": rpc error: code = NotFound desc = could not find container \"7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770\": container with ID starting with 7e96cd076242b51027a8e14263c006a1985273a83049cb30c180a3e21c79d770 not found: ID does not exist"
Jul 01 14:59:24.791785 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.791764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"]
Jul 01 14:59:24.796398 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:24.796376    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-665mj"]
Jul 01 14:59:25.957175 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:25.957140    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" path="/var/lib/kubelet/pods/3b88f574-0d17-4647-b5ab-26a607f24c9c/volumes"
Jul 01 14:59:28.618651 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618614    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"]
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618931    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="storage-initializer"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="storage-initializer"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618958    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kserve-container"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618964    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kserve-container"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618972    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kube-rbac-proxy"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.618978    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kube-rbac-proxy"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.619026    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kserve-container"
Jul 01 14:59:28.619056 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.619034    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b88f574-0d17-4647-b5ab-26a607f24c9c" containerName="kube-rbac-proxy"
Jul 01 14:59:28.621249 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.621231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.624223 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.624198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 14:59:28.624358 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.624254    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 01 14:59:28.624358 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.624258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 14:59:28.625411 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.625395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 14:59:28.625645 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.625627    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 01 14:59:28.625706 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.625646    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 14:59:28.625706 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.625676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4cvh9\""
Jul 01 14:59:28.633550 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.633529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"]
Jul 01 14:59:28.691470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.691429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.691470 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.691473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftvt\" (UniqueName: \"kubernetes.io/projected/ee800482-a912-49c3-bcb8-248c50e015d6-kube-api-access-lftvt\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.691732 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.691541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.691732 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.691594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee800482-a912-49c3-bcb8-248c50e015d6-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.691732 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.691628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee800482-a912-49c3-bcb8-248c50e015d6-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.792363 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.792326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee800482-a912-49c3-bcb8-248c50e015d6-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.792614 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.792413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.792614 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.792453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftvt\" (UniqueName: \"kubernetes.io/projected/ee800482-a912-49c3-bcb8-248c50e015d6-kube-api-access-lftvt\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.792614 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.792487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.792614 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.792513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee800482-a912-49c3-bcb8-248c50e015d6-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.792911 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.792882    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee800482-a912-49c3-bcb8-248c50e015d6-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.793142 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.793122    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.793219 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.793179    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.794763 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.794742    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee800482-a912-49c3-bcb8-248c50e015d6-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.802877 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.802853    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftvt\" (UniqueName: \"kubernetes.io/projected/ee800482-a912-49c3-bcb8-248c50e015d6-kube-api-access-lftvt\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:28.931277 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:28.931242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:29.065894 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:29.065869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"]
Jul 01 14:59:29.067892 ip-10-0-128-59 kubenswrapper[2579]: W0701 14:59:29.067862    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee800482_a912_49c3_bcb8_248c50e015d6.slice/crio-bf3fae4aaf006e601582a5d73384c9090d406616d30eb25637c702e5dfd79266 WatchSource:0}: Error finding container bf3fae4aaf006e601582a5d73384c9090d406616d30eb25637c702e5dfd79266: Status 404 returned error can't find the container with id bf3fae4aaf006e601582a5d73384c9090d406616d30eb25637c702e5dfd79266
Jul 01 14:59:29.788602 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:29.788569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerStarted","Data":"9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0"}
Jul 01 14:59:29.789010 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:29.788631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerStarted","Data":"bf3fae4aaf006e601582a5d73384c9090d406616d30eb25637c702e5dfd79266"}
Jul 01 14:59:47.879101 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:47.879064    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee800482-a912-49c3-bcb8-248c50e015d6" containerID="9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0" exitCode=0
Jul 01 14:59:47.879593 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:47.879140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerDied","Data":"9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0"}
Jul 01 14:59:48.886750 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:48.886717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerStarted","Data":"9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424"}
Jul 01 14:59:48.887168 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:48.886779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerStarted","Data":"e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f"}
Jul 01 14:59:48.887168 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:48.887005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:48.887168 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:48.887164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 14:59:48.909439 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:48.909372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" podStartSLOduration=2.244233957 podStartE2EDuration="20.909350273s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.665116316s" isStatefulPod=false podCreationTimestamp="2026-07-01 14:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 14:59:48.908098576 +0000 UTC m=+1267.434382581" watchObservedRunningTime="2026-07-01 14:59:48.909350273 +0000 UTC m=+1267.435634277"
Jul 01 14:59:54.896883 ip-10-0-128-59 kubenswrapper[2579]: I0701 14:59:54.896854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 15:01:18.955693 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:18.955654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 15:01:28.827137 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:28.827103    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"]
Jul 01 15:01:28.827588 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:28.827356    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kserve-container" containerID="cri-o://e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f" gracePeriod=30
Jul 01 15:01:28.827588 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:28.827439    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kube-rbac-proxy" containerID="cri-o://9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424" gracePeriod=30
Jul 01 15:01:28.952482 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:28.952449    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 01 15:01:29.347447 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:29.347412    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee800482-a912-49c3-bcb8-248c50e015d6" containerID="9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424" exitCode=2
Jul 01 15:01:29.347629 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:29.347486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerDied","Data":"9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424"}
Jul 01 15:01:29.881088 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:29.881059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 15:01:30.025693 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.025653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lftvt\" (UniqueName: \"kubernetes.io/projected/ee800482-a912-49c3-bcb8-248c50e015d6-kube-api-access-lftvt\") pod \"ee800482-a912-49c3-bcb8-248c50e015d6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") "
Jul 01 15:01:30.025903 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.025716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee800482-a912-49c3-bcb8-248c50e015d6-kserve-provision-location\") pod \"ee800482-a912-49c3-bcb8-248c50e015d6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") "
Jul 01 15:01:30.025903 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.025758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee800482-a912-49c3-bcb8-248c50e015d6-proxy-tls\") pod \"ee800482-a912-49c3-bcb8-248c50e015d6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") "
Jul 01 15:01:30.026039 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.026010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-cabundle-cert\") pod \"ee800482-a912-49c3-bcb8-248c50e015d6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") "
Jul 01 15:01:30.026098 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.026054    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-kube-rbac-proxy-sar-config\") pod \"ee800482-a912-49c3-bcb8-248c50e015d6\" (UID: \"ee800482-a912-49c3-bcb8-248c50e015d6\") "
Jul 01 15:01:30.026430 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.026402    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee800482-a912-49c3-bcb8-248c50e015d6" (UID: "ee800482-a912-49c3-bcb8-248c50e015d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:01:30.026522 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.026477    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee800482-a912-49c3-bcb8-248c50e015d6" (UID: "ee800482-a912-49c3-bcb8-248c50e015d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:01:30.027981 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.027952    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee800482-a912-49c3-bcb8-248c50e015d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee800482-a912-49c3-bcb8-248c50e015d6" (UID: "ee800482-a912-49c3-bcb8-248c50e015d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 15:01:30.027981 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.027965    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee800482-a912-49c3-bcb8-248c50e015d6-kube-api-access-lftvt" (OuterVolumeSpecName: "kube-api-access-lftvt") pod "ee800482-a912-49c3-bcb8-248c50e015d6" (UID: "ee800482-a912-49c3-bcb8-248c50e015d6"). InnerVolumeSpecName "kube-api-access-lftvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 15:01:30.030648 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.030626    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee800482-a912-49c3-bcb8-248c50e015d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee800482-a912-49c3-bcb8-248c50e015d6" (UID: "ee800482-a912-49c3-bcb8-248c50e015d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 15:01:30.127069 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.127035    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lftvt\" (UniqueName: \"kubernetes.io/projected/ee800482-a912-49c3-bcb8-248c50e015d6-kube-api-access-lftvt\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:01:30.127069 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.127062    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee800482-a912-49c3-bcb8-248c50e015d6-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:01:30.127069 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.127072    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee800482-a912-49c3-bcb8-248c50e015d6-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:01:30.127069 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.127082    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:01:30.127378 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.127091    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee800482-a912-49c3-bcb8-248c50e015d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:01:30.353633 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.353545    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee800482-a912-49c3-bcb8-248c50e015d6" containerID="e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f" exitCode=0
Jul 01 15:01:30.353633 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.353591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerDied","Data":"e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f"}
Jul 01 15:01:30.353633 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.353613    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"
Jul 01 15:01:30.353850 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.353652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6" event={"ID":"ee800482-a912-49c3-bcb8-248c50e015d6","Type":"ContainerDied","Data":"bf3fae4aaf006e601582a5d73384c9090d406616d30eb25637c702e5dfd79266"}
Jul 01 15:01:30.353850 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.353691    2579 scope.go:117] "RemoveContainer" containerID="9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424"
Jul 01 15:01:30.362700 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.362680    2579 scope.go:117] "RemoveContainer" containerID="e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f"
Jul 01 15:01:30.370570 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.370550    2579 scope.go:117] "RemoveContainer" containerID="9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0"
Jul 01 15:01:30.377995 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.377970    2579 scope.go:117] "RemoveContainer" containerID="9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424"
Jul 01 15:01:30.378275 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:01:30.378253    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424\": container with ID starting with 9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424 not found: ID does not exist" containerID="9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424"
Jul 01 15:01:30.378391 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.378289    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424"} err="failed to get container status \"9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424\": rpc error: code = NotFound desc = could not find container \"9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424\": container with ID starting with 9c61a1826dc11d12863fd097f8b7d7b8d46994a5384939a41a9fc1d3a2fcb424 not found: ID does not exist"
Jul 01 15:01:30.378391 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.378331    2579 scope.go:117] "RemoveContainer" containerID="e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f"
Jul 01 15:01:30.378595 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:01:30.378576    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f\": container with ID starting with e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f not found: ID does not exist" containerID="e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f"
Jul 01 15:01:30.378651 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.378604    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f"} err="failed to get container status \"e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f\": rpc error: code = NotFound desc = could not find container \"e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f\": container with ID starting with e0c27869890404431bde5ee1337d6659f2fba2d7f5e4690cabfc2df2c1442f0f not found: ID does not exist"
Jul 01 15:01:30.378651 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.378619    2579 scope.go:117] "RemoveContainer" containerID="9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0"
Jul 01 15:01:30.378872 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:01:30.378856    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0\": container with ID starting with 9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0 not found: ID does not exist" containerID="9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0"
Jul 01 15:01:30.378909 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.378886    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0"} err="failed to get container status \"9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0\": rpc error: code = NotFound desc = could not find container \"9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0\": container with ID starting with 9f20baa25f37ff393fc6432457d7936f8c2e7edef82363d62751e6ab429549d0 not found: ID does not exist"
Jul 01 15:01:30.383692 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.383673    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"]
Jul 01 15:01:30.390087 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:30.390065    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-5bdl6"]
Jul 01 15:01:31.957582 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:31.957535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" path="/var/lib/kubelet/pods/ee800482-a912-49c3-bcb8-248c50e015d6/volumes"
Jul 01 15:01:33.940900 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.940848    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"]
Jul 01 15:01:33.941297 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941183    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kserve-container"
Jul 01 15:01:33.941297 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941195    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kserve-container"
Jul 01 15:01:33.941297 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="storage-initializer"
Jul 01 15:01:33.941297 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="storage-initializer"
Jul 01 15:01:33.941297 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941235    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kube-rbac-proxy"
Jul 01 15:01:33.941297 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941241    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kube-rbac-proxy"
Jul 01 15:01:33.941535 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941320    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kserve-container"
Jul 01 15:01:33.941535 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.941334    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee800482-a912-49c3-bcb8-248c50e015d6" containerName="kube-rbac-proxy"
Jul 01 15:01:33.946362 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.946339    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:33.948897 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.948849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 01 15:01:33.948897 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.948885    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4cvh9\""
Jul 01 15:01:33.949064 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.949024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 15:01:33.949146 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.949127    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 15:01:33.949715 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.949697    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 15:01:33.949815 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.949714    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 15:01:33.949815 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.949700    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 01 15:01:33.954956 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.954930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:33.955138 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.955113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq5nm\" (UniqueName: \"kubernetes.io/projected/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-api-access-mq5nm\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:33.955251 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.955179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:33.955251 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.955235    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:33.955387 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.955341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:33.957994 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:33.957975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"]
Jul 01 15:01:34.056667 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.056621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.056667 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.056674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq5nm\" (UniqueName: \"kubernetes.io/projected/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-api-access-mq5nm\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.056903 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.056704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.056903 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.056740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.056903 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.056796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.057135 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:01:34.056965    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 01 15:01:34.057135 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:01:34.057059    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls podName:245d43b2-1308-4e9a-96c8-d57e47a26dfa nodeName:}" failed. No retries permitted until 2026-07-01 15:01:34.557033751 +0000 UTC m=+1373.083317738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" (UID: "245d43b2-1308-4e9a-96c8-d57e47a26dfa") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 01 15:01:34.057135 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.057078    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.057520 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.057499    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.057570 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.057541    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.066385 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.066366    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq5nm\" (UniqueName: \"kubernetes.io/projected/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-api-access-mq5nm\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.561539 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.561500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.563895 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.563875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.857731 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.857629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:34.995575 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:34.995459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"]
Jul 01 15:01:34.997823 ip-10-0-128-59 kubenswrapper[2579]: W0701 15:01:34.997792    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245d43b2_1308_4e9a_96c8_d57e47a26dfa.slice/crio-f6e2dbd118078c4a390cc1da5c502106e4abffcd9244f5de4c31662b905a2e3d WatchSource:0}: Error finding container f6e2dbd118078c4a390cc1da5c502106e4abffcd9244f5de4c31662b905a2e3d: Status 404 returned error can't find the container with id f6e2dbd118078c4a390cc1da5c502106e4abffcd9244f5de4c31662b905a2e3d
Jul 01 15:01:35.378262 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:35.378222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerStarted","Data":"edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e"}
Jul 01 15:01:35.378474 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:35.378295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerStarted","Data":"f6e2dbd118078c4a390cc1da5c502106e4abffcd9244f5de4c31662b905a2e3d"}
Jul 01 15:01:40.407853 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:40.407819    2579 generic.go:356] "Generic (PLEG): container finished" podID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerID="edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e" exitCode=0
Jul 01 15:01:40.408365 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:40.407896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerDied","Data":"edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e"}
Jul 01 15:01:41.414812 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:41.414783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerStarted","Data":"3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72"}
Jul 01 15:01:41.415291 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:41.414848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerStarted","Data":"6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed"}
Jul 01 15:01:41.415291 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:41.415062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:41.438428 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:41.438368    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" podStartSLOduration=3.549710197 podStartE2EDuration="8.43834665s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.888636453s" isStatefulPod=false podCreationTimestamp="2026-07-01 15:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:01:41.436437273 +0000 UTC m=+1379.962721298" watchObservedRunningTime="2026-07-01 15:01:41.43834665 +0000 UTC m=+1379.964630656"
Jul 01 15:01:42.419055 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:42.419014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:01:48.433438 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:01:48.433409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:03:11.959050 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:11.959015    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:03:14.072689 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:14.072657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"]
Jul 01 15:03:14.073124 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:14.072916    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kserve-container" containerID="cri-o://6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed" gracePeriod=30
Jul 01 15:03:14.073124 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:14.072936    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kube-rbac-proxy" containerID="cri-o://3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72" gracePeriod=30
Jul 01 15:03:14.854813 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:14.854781    2579 generic.go:356] "Generic (PLEG): container finished" podID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerID="3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72" exitCode=2
Jul 01 15:03:14.854938 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:14.854847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerDied","Data":"3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72"}
Jul 01 15:03:15.022582 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.022556    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:03:15.126382 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-cabundle-cert\") pod \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") "
Jul 01 15:03:15.126382 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-rbac-proxy-sar-config\") pod \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") "
Jul 01 15:03:15.126915 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls\") pod \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") "
Jul 01 15:03:15.126915 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq5nm\" (UniqueName: \"kubernetes.io/projected/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-api-access-mq5nm\") pod \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") "
Jul 01 15:03:15.126915 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kserve-provision-location\") pod \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\" (UID: \"245d43b2-1308-4e9a-96c8-d57e47a26dfa\") "
Jul 01 15:03:15.126915 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126707    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "245d43b2-1308-4e9a-96c8-d57e47a26dfa" (UID: "245d43b2-1308-4e9a-96c8-d57e47a26dfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:03:15.126915 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.126839    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:15.127276 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.127246    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "245d43b2-1308-4e9a-96c8-d57e47a26dfa" (UID: "245d43b2-1308-4e9a-96c8-d57e47a26dfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:03:15.129149 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.129107    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-api-access-mq5nm" (OuterVolumeSpecName: "kube-api-access-mq5nm") pod "245d43b2-1308-4e9a-96c8-d57e47a26dfa" (UID: "245d43b2-1308-4e9a-96c8-d57e47a26dfa"). InnerVolumeSpecName "kube-api-access-mq5nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 15:03:15.129394 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.129370    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "245d43b2-1308-4e9a-96c8-d57e47a26dfa" (UID: "245d43b2-1308-4e9a-96c8-d57e47a26dfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 15:03:15.133281 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.133246    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "245d43b2-1308-4e9a-96c8-d57e47a26dfa" (UID: "245d43b2-1308-4e9a-96c8-d57e47a26dfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 15:03:15.228017 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.227984    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245d43b2-1308-4e9a-96c8-d57e47a26dfa-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:15.228017 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.228015    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq5nm\" (UniqueName: \"kubernetes.io/projected/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-api-access-mq5nm\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:15.228017 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.228025    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:15.228272 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.228033    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245d43b2-1308-4e9a-96c8-d57e47a26dfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:15.861351 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.861291    2579 generic.go:356] "Generic (PLEG): container finished" podID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerID="6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed" exitCode=0
Jul 01 15:03:15.861565 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.861372    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"
Jul 01 15:03:15.861565 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.861376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerDied","Data":"6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed"}
Jul 01 15:03:15.861565 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.861451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655" event={"ID":"245d43b2-1308-4e9a-96c8-d57e47a26dfa","Type":"ContainerDied","Data":"f6e2dbd118078c4a390cc1da5c502106e4abffcd9244f5de4c31662b905a2e3d"}
Jul 01 15:03:15.861565 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.861488    2579 scope.go:117] "RemoveContainer" containerID="3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72"
Jul 01 15:03:15.870350 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.870333    2579 scope.go:117] "RemoveContainer" containerID="6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed"
Jul 01 15:03:15.877772 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.877755    2579 scope.go:117] "RemoveContainer" containerID="edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e"
Jul 01 15:03:15.883856 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.883835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"]
Jul 01 15:03:15.885230 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.885212    2579 scope.go:117] "RemoveContainer" containerID="3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72"
Jul 01 15:03:15.885525 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:03:15.885506    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72\": container with ID starting with 3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72 not found: ID does not exist" containerID="3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72"
Jul 01 15:03:15.885612 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.885538    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72"} err="failed to get container status \"3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72\": rpc error: code = NotFound desc = could not find container \"3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72\": container with ID starting with 3bdf3ebe20bd33027f8004f88d9cbe2b2f6e674ed509d5b8567661c137f13f72 not found: ID does not exist"
Jul 01 15:03:15.885612 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.885563    2579 scope.go:117] "RemoveContainer" containerID="6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed"
Jul 01 15:03:15.885855 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:03:15.885838    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed\": container with ID starting with 6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed not found: ID does not exist" containerID="6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed"
Jul 01 15:03:15.885900 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.885861    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed"} err="failed to get container status \"6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed\": rpc error: code = NotFound desc = could not find container \"6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed\": container with ID starting with 6900b4c41f3e3455d2a845de01015c3bcab3a334aee401e2dfa4ba8d36004fed not found: ID does not exist"
Jul 01 15:03:15.885900 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.885878    2579 scope.go:117] "RemoveContainer" containerID="edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e"
Jul 01 15:03:15.886127 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:03:15.886103    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e\": container with ID starting with edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e not found: ID does not exist" containerID="edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e"
Jul 01 15:03:15.886197 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.886137    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e"} err="failed to get container status \"edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e\": rpc error: code = NotFound desc = could not find container \"edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e\": container with ID starting with edd4b7782939c7441e39de63a5ac7a708fd9f71081ea93f6c00bc20e0473b77e not found: ID does not exist"
Jul 01 15:03:15.888430 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.888410    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-hm655"]
Jul 01 15:03:15.956863 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:15.956832    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" path="/var/lib/kubelet/pods/245d43b2-1308-4e9a-96c8-d57e47a26dfa/volumes"
Jul 01 15:03:19.262506 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262467    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"]
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262786    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="storage-initializer"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262797    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="storage-initializer"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262807    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kserve-container"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262813    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kserve-container"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262836    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kube-rbac-proxy"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262842    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kube-rbac-proxy"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262894    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kserve-container"
Jul 01 15:03:19.262907 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.262902    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="245d43b2-1308-4e9a-96c8-d57e47a26dfa" containerName="kube-rbac-proxy"
Jul 01 15:03:19.267717 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.267698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.270832 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.270803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 01 15:03:19.271138 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.271117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 15:03:19.271248 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.271183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 01 15:03:19.271248 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.271187    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 15:03:19.271398 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.271350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4cvh9\""
Jul 01 15:03:19.271586 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.271573    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 15:03:19.271669 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.271651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 15:03:19.281002 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.280975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"]
Jul 01 15:03:19.467648 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.467605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c542e3-6703-4969-8f2c-f44418e596e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.467648 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.467655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c542e3-6703-4969-8f2c-f44418e596e4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.467888 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.467673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.467888 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.467732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkc7\" (UniqueName: \"kubernetes.io/projected/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-api-access-2jkc7\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.467888 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.467800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.568535 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.568438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c542e3-6703-4969-8f2c-f44418e596e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.568535 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.568497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c542e3-6703-4969-8f2c-f44418e596e4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.568535 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.568520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.568852 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.568545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkc7\" (UniqueName: \"kubernetes.io/projected/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-api-access-2jkc7\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.568852 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.568610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.568981 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.568886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c542e3-6703-4969-8f2c-f44418e596e4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.569322 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.569283    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.569379 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.569338    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.571020 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.570997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c542e3-6703-4969-8f2c-f44418e596e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.578324 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.578282    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkc7\" (UniqueName: \"kubernetes.io/projected/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-api-access-2jkc7\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.578455 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.578443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:19.715280 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.715253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"]
Jul 01 15:03:19.717713 ip-10-0-128-59 kubenswrapper[2579]: W0701 15:03:19.717678    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c542e3_6703_4969_8f2c_f44418e596e4.slice/crio-171a49ce5be264467c5da58c9214ae6d710ff4c35448d79c6bb0919f30863c20 WatchSource:0}: Error finding container 171a49ce5be264467c5da58c9214ae6d710ff4c35448d79c6bb0919f30863c20: Status 404 returned error can't find the container with id 171a49ce5be264467c5da58c9214ae6d710ff4c35448d79c6bb0919f30863c20
Jul 01 15:03:19.883479 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.883384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerStarted","Data":"dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5"}
Jul 01 15:03:19.883479 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:19.883465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerStarted","Data":"171a49ce5be264467c5da58c9214ae6d710ff4c35448d79c6bb0919f30863c20"}
Jul 01 15:03:28.927166 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:28.927130    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerID="dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5" exitCode=0
Jul 01 15:03:28.927572 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:28.927170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerDied","Data":"dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5"}
Jul 01 15:03:29.942079 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:29.942044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerStarted","Data":"f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e"}
Jul 01 15:03:29.942474 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:29.942138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerStarted","Data":"660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326"}
Jul 01 15:03:29.942474 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:29.942355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:29.964940 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:29.964880    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" podStartSLOduration=2.118807913 podStartE2EDuration="10.964865504s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.846057591s" isStatefulPod=false podCreationTimestamp="2026-07-01 15:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:03:29.962864653 +0000 UTC m=+1488.489148676" watchObservedRunningTime="2026-07-01 15:03:29.964865504 +0000 UTC m=+1488.491149507"
Jul 01 15:03:30.948256 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:30.948223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:30.949315 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:30.949274    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 15:03:31.954824 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:31.954781    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 15:03:36.959596 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:36.959570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:36.960155 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:36.960118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 01 15:03:42.031995 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:42.031968    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 15:03:42.032474 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:42.032422    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 15:03:46.961469 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:46.961439    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:49.306941 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:49.306909    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"]
Jul 01 15:03:49.307361 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:49.307151    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container" containerID="cri-o://660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326" gracePeriod=30
Jul 01 15:03:49.307361 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:49.307161    2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kube-rbac-proxy" containerID="cri-o://f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e" gracePeriod=30
Jul 01 15:03:50.043615 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.043585    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerID="f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e" exitCode=2
Jul 01 15:03:50.043727 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.043648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerDied","Data":"f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e"}
Jul 01 15:03:50.259616 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.259590    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:50.343052 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.342949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c542e3-6703-4969-8f2c-f44418e596e4-proxy-tls\") pod \"d6c542e3-6703-4969-8f2c-f44418e596e4\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") "
Jul 01 15:03:50.343052 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.342989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-cabundle-cert\") pod \"d6c542e3-6703-4969-8f2c-f44418e596e4\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") "
Jul 01 15:03:50.343052 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.343013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c542e3-6703-4969-8f2c-f44418e596e4-kserve-provision-location\") pod \"d6c542e3-6703-4969-8f2c-f44418e596e4\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") "
Jul 01 15:03:50.343618 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.343068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jkc7\" (UniqueName: \"kubernetes.io/projected/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-api-access-2jkc7\") pod \"d6c542e3-6703-4969-8f2c-f44418e596e4\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") "
Jul 01 15:03:50.343618 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.343146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-rbac-proxy-sar-config\") pod \"d6c542e3-6703-4969-8f2c-f44418e596e4\" (UID: \"d6c542e3-6703-4969-8f2c-f44418e596e4\") "
Jul 01 15:03:50.343618 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.343502    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6c542e3-6703-4969-8f2c-f44418e596e4" (UID: "d6c542e3-6703-4969-8f2c-f44418e596e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:03:50.343618 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.343507    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6c542e3-6703-4969-8f2c-f44418e596e4" (UID: "d6c542e3-6703-4969-8f2c-f44418e596e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 15:03:50.344069 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.344050    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c542e3-6703-4969-8f2c-f44418e596e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6c542e3-6703-4969-8f2c-f44418e596e4" (UID: "d6c542e3-6703-4969-8f2c-f44418e596e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 15:03:50.345253 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.345231    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c542e3-6703-4969-8f2c-f44418e596e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6c542e3-6703-4969-8f2c-f44418e596e4" (UID: "d6c542e3-6703-4969-8f2c-f44418e596e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 15:03:50.345343 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.345278    2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-api-access-2jkc7" (OuterVolumeSpecName: "kube-api-access-2jkc7") pod "d6c542e3-6703-4969-8f2c-f44418e596e4" (UID: "d6c542e3-6703-4969-8f2c-f44418e596e4"). InnerVolumeSpecName "kube-api-access-2jkc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 15:03:50.444294 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.444255    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6c542e3-6703-4969-8f2c-f44418e596e4-proxy-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:50.444294 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.444286    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-cabundle-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:50.444294 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.444297    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c542e3-6703-4969-8f2c-f44418e596e4-kserve-provision-location\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:50.444553 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.444334    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jkc7\" (UniqueName: \"kubernetes.io/projected/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-api-access-2jkc7\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:50.444553 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:50.444346    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6c542e3-6703-4969-8f2c-f44418e596e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\""
Jul 01 15:03:51.050268 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.050235    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerID="660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326" exitCode=0
Jul 01 15:03:51.050464 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.050293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"
Jul 01 15:03:51.050464 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.050325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerDied","Data":"660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326"}
Jul 01 15:03:51.050464 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.050383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm" event={"ID":"d6c542e3-6703-4969-8f2c-f44418e596e4","Type":"ContainerDied","Data":"171a49ce5be264467c5da58c9214ae6d710ff4c35448d79c6bb0919f30863c20"}
Jul 01 15:03:51.050464 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.050408    2579 scope.go:117] "RemoveContainer" containerID="f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e"
Jul 01 15:03:51.059138 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.059122    2579 scope.go:117] "RemoveContainer" containerID="660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326"
Jul 01 15:03:51.066478 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.066462    2579 scope.go:117] "RemoveContainer" containerID="dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5"
Jul 01 15:03:51.073787 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.073764    2579 scope.go:117] "RemoveContainer" containerID="f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e"
Jul 01 15:03:51.074106 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:03:51.074087    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e\": container with ID starting with f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e not found: ID does not exist" containerID="f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e"
Jul 01 15:03:51.074165 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.074114    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e"} err="failed to get container status \"f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e\": rpc error: code = NotFound desc = could not find container \"f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e\": container with ID starting with f401a711a0b50acd250c941a36c8c5353d416a3a1f76c733ddcf918727cccb9e not found: ID does not exist"
Jul 01 15:03:51.074165 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.074130    2579 scope.go:117] "RemoveContainer" containerID="660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326"
Jul 01 15:03:51.074459 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:03:51.074439    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326\": container with ID starting with 660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326 not found: ID does not exist" containerID="660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326"
Jul 01 15:03:51.074526 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.074460    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326"} err="failed to get container status \"660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326\": rpc error: code = NotFound desc = could not find container \"660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326\": container with ID starting with 660a94c4e27f39d65dcf82b25d4b36e8e47dca9956f765e0266561c3994d4326 not found: ID does not exist"
Jul 01 15:03:51.074526 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.074474    2579 scope.go:117] "RemoveContainer" containerID="dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5"
Jul 01 15:03:51.074670 ip-10-0-128-59 kubenswrapper[2579]: E0701 15:03:51.074656    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5\": container with ID starting with dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5 not found: ID does not exist" containerID="dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5"
Jul 01 15:03:51.074710 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.074673    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5"} err="failed to get container status \"dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5\": rpc error: code = NotFound desc = could not find container \"dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5\": container with ID starting with dceef12fc80d585f3f49eca3589737955c3a288fcc571a6234865fbff91737a5 not found: ID does not exist"
Jul 01 15:03:51.075589 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.075570    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"]
Jul 01 15:03:51.081208 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.081185    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-9twwm"]
Jul 01 15:03:51.957493 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:03:51.957454    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" path="/var/lib/kubelet/pods/d6c542e3-6703-4969-8f2c-f44418e596e4/volumes"
Jul 01 15:04:25.628996 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.628954    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cdwxh/must-gather-8s767"]
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629444    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629462    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629476    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kube-rbac-proxy"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629485    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kube-rbac-proxy"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629521    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="storage-initializer"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629530    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="storage-initializer"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629624    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kserve-container"
Jul 01 15:04:25.629665 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.629636    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c542e3-6703-4969-8f2c-f44418e596e4" containerName="kube-rbac-proxy"
Jul 01 15:04:25.633035 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.633013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.635425 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.635399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cdwxh\"/\"openshift-service-ca.crt\""
Jul 01 15:04:25.636225 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.636204    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cdwxh\"/\"kube-root-ca.crt\""
Jul 01 15:04:25.636340 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.636239    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cdwxh\"/\"default-dockercfg-km4rb\""
Jul 01 15:04:25.641008 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.640985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdwxh/must-gather-8s767"]
Jul 01 15:04:25.654342 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.654312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v96p\" (UniqueName: \"kubernetes.io/projected/9e404147-5d59-4a6f-89aa-7c15d28f8307-kube-api-access-5v96p\") pod \"must-gather-8s767\" (UID: \"9e404147-5d59-4a6f-89aa-7c15d28f8307\") " pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.654467 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.654393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e404147-5d59-4a6f-89aa-7c15d28f8307-must-gather-output\") pod \"must-gather-8s767\" (UID: \"9e404147-5d59-4a6f-89aa-7c15d28f8307\") " pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.755710 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.755673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e404147-5d59-4a6f-89aa-7c15d28f8307-must-gather-output\") pod \"must-gather-8s767\" (UID: \"9e404147-5d59-4a6f-89aa-7c15d28f8307\") " pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.755889 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.755818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v96p\" (UniqueName: \"kubernetes.io/projected/9e404147-5d59-4a6f-89aa-7c15d28f8307-kube-api-access-5v96p\") pod \"must-gather-8s767\" (UID: \"9e404147-5d59-4a6f-89aa-7c15d28f8307\") " pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.756049 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.756025    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e404147-5d59-4a6f-89aa-7c15d28f8307-must-gather-output\") pod \"must-gather-8s767\" (UID: \"9e404147-5d59-4a6f-89aa-7c15d28f8307\") " pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.764986 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.764956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v96p\" (UniqueName: \"kubernetes.io/projected/9e404147-5d59-4a6f-89aa-7c15d28f8307-kube-api-access-5v96p\") pod \"must-gather-8s767\" (UID: \"9e404147-5d59-4a6f-89aa-7c15d28f8307\") " pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:25.952865 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:25.952832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdwxh/must-gather-8s767"
Jul 01 15:04:26.084454 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:26.084420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdwxh/must-gather-8s767"]
Jul 01 15:04:26.087595 ip-10-0-128-59 kubenswrapper[2579]: W0701 15:04:26.087565    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e404147_5d59_4a6f_89aa_7c15d28f8307.slice/crio-ebea4c6d0de8320217478d690f11b6cd91e6e6839e3e8f44992216114844960b WatchSource:0}: Error finding container ebea4c6d0de8320217478d690f11b6cd91e6e6839e3e8f44992216114844960b: Status 404 returned error can't find the container with id ebea4c6d0de8320217478d690f11b6cd91e6e6839e3e8f44992216114844960b
Jul 01 15:04:26.089360 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:26.089341    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 15:04:26.220699 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:26.220606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdwxh/must-gather-8s767" event={"ID":"9e404147-5d59-4a6f-89aa-7c15d28f8307","Type":"ContainerStarted","Data":"ebea4c6d0de8320217478d690f11b6cd91e6e6839e3e8f44992216114844960b"}
Jul 01 15:04:29.238207 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:29.238114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdwxh/must-gather-8s767" event={"ID":"9e404147-5d59-4a6f-89aa-7c15d28f8307","Type":"ContainerStarted","Data":"7fad1b510374ed1901ceba88dd7d4fb583e24b587918bcac82c9354da6e3f9af"}
Jul 01 15:04:29.238207 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:29.238168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdwxh/must-gather-8s767" event={"ID":"9e404147-5d59-4a6f-89aa-7c15d28f8307","Type":"ContainerStarted","Data":"9a016f5ce8a3e96397663392abb2ff242b03130328877c4101d467b8c32088b7"}
Jul 01 15:04:29.256087 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:29.256036    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cdwxh/must-gather-8s767" podStartSLOduration=2.078601678 podStartE2EDuration="4.256021712s" totalImagesPullingTime="2.177420034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 15:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:04:29.254489237 +0000 UTC m=+1547.780773243" watchObservedRunningTime="2026-07-01 15:04:29.256021712 +0000 UTC m=+1547.782305774"
Jul 01 15:04:30.109169 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:30.109139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xl6ft_7c48eacc-cd11-4893-94ef-d8368f954658/global-pull-secret-syncer/0.log"
Jul 01 15:04:30.191954 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:30.191925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wdmbn_c02426cb-5b55-4f73-ab60-16df7c4dcc2a/konnectivity-agent/0.log"
Jul 01 15:04:30.257282 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:30.257245    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-59.ec2.internal_371354312e904fda882e14f06ac6f309/haproxy/0.log"
Jul 01 15:04:34.168398 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.168329    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/alertmanager/0.log"
Jul 01 15:04:34.210093 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.210063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/config-reloader/0.log"
Jul 01 15:04:34.244118 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.244085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/kube-rbac-proxy-web/0.log"
Jul 01 15:04:34.281597 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.281555    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/kube-rbac-proxy/0.log"
Jul 01 15:04:34.308878 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.308854    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/kube-rbac-proxy-metric/0.log"
Jul 01 15:04:34.354009 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.353799    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/prom-label-proxy/0.log"
Jul 01 15:04:34.389252 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.389223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f3bec10-78d2-4de0-803e-a844b37c72cb/init-config-reloader/0.log"
Jul 01 15:04:34.442985 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.442435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-s4fnf_e2242c5d-0e33-4904-bd89-024c85303ff3/cluster-monitoring-operator/0.log"
Jul 01 15:04:34.575197 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.575140    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85459649d6-n7jdh_f8c25f9b-945f-4acb-a4f5-523f0d2fb0de/metrics-server/0.log"
Jul 01 15:04:34.614401 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.614357    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-48zcd_ba8fd0f3-7b89-4e23-a6c1-797c6cb337cc/monitoring-plugin/0.log"
Jul 01 15:04:34.765794 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.765702    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9q4n_dc82c488-6538-464d-947e-24f2151a3836/node-exporter/0.log"
Jul 01 15:04:34.792205 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.792128    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9q4n_dc82c488-6538-464d-947e-24f2151a3836/kube-rbac-proxy/0.log"
Jul 01 15:04:34.823293 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.823267    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9q4n_dc82c488-6538-464d-947e-24f2151a3836/init-textfile/0.log"
Jul 01 15:04:34.938704 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.938667    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-fkvt2_d59ad5a3-db43-4872-99ba-ea96be6713f5/kube-rbac-proxy-main/0.log"
Jul 01 15:04:34.969243 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:34.969212    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-fkvt2_d59ad5a3-db43-4872-99ba-ea96be6713f5/kube-rbac-proxy-self/0.log"
Jul 01 15:04:35.003243 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.003214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-fkvt2_d59ad5a3-db43-4872-99ba-ea96be6713f5/openshift-state-metrics/0.log"
Jul 01 15:04:35.054477 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.054385    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/prometheus/0.log"
Jul 01 15:04:35.076572 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.076493    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/config-reloader/0.log"
Jul 01 15:04:35.104393 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.104289    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/thanos-sidecar/0.log"
Jul 01 15:04:35.132510 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.132484    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/kube-rbac-proxy-web/0.log"
Jul 01 15:04:35.159775 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.159745    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/kube-rbac-proxy/0.log"
Jul 01 15:04:35.188644 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.188613    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/kube-rbac-proxy-thanos/0.log"
Jul 01 15:04:35.216728 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.216698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_41bf6482-4ee4-42e3-bfab-a0d6c74959a5/init-config-reloader/0.log"
Jul 01 15:04:35.266442 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.266392    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-td7l8_d8e7846f-bae0-4490-bc43-cebd44e2ccda/prometheus-operator/0.log"
Jul 01 15:04:35.290524 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.290473    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-td7l8_d8e7846f-bae0-4490-bc43-cebd44e2ccda/kube-rbac-proxy/0.log"
Jul 01 15:04:35.329770 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:35.329742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-vsds7_b1aa3403-9b84-48e0-9d3b-fc674aa4af55/prometheus-operator-admission-webhook/0.log"
Jul 01 15:04:38.191116 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:38.191035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-rmtc6_7b6625e6-3a8f-4859-b5aa-256c707589d2/download-server/0.log"
Jul 01 15:04:38.778826 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:38.778795    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-p4zfr_5595e14a-62bd-47b1-b12e-125d7ecbd24d/volume-data-source-validator/0.log"
Jul 01 15:04:39.432648 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.432606    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"]
Jul 01 15:04:39.438181 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.438146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.457144 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.457112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"]
Jul 01 15:04:39.599356 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.599322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-lib-modules\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.599356 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.599365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-podres\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.599596 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.599382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpv2\" (UniqueName: \"kubernetes.io/projected/bf4622f6-b68a-4f35-8f58-3c5d459103c2-kube-api-access-ffpv2\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.599596 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.599471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-proc\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.599596 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.599528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-sys\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.700264 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-proc\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.700621 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-sys\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.700716 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-lib-modules\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.700779 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-podres\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.700779 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpv2\" (UniqueName: \"kubernetes.io/projected/bf4622f6-b68a-4f35-8f58-3c5d459103c2-kube-api-access-ffpv2\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.700967 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-sys\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.701194 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.701175    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-lib-modules\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.701348 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.701325    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-podres\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.701431 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.700423    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf4622f6-b68a-4f35-8f58-3c5d459103c2-proc\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.711676 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.711641    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpv2\" (UniqueName: \"kubernetes.io/projected/bf4622f6-b68a-4f35-8f58-3c5d459103c2-kube-api-access-ffpv2\") pod \"perf-node-gather-daemonset-9hl99\" (UID: \"bf4622f6-b68a-4f35-8f58-3c5d459103c2\") " pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.753853 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.753814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:39.934259 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:39.934228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"]
Jul 01 15:04:39.939053 ip-10-0-128-59 kubenswrapper[2579]: W0701 15:04:39.939012    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf4622f6_b68a_4f35_8f58_3c5d459103c2.slice/crio-2b8e640b8cf79124d5304ffff22cf95b0bde0d8f88644d8d51c874a4ae20a274 WatchSource:0}: Error finding container 2b8e640b8cf79124d5304ffff22cf95b0bde0d8f88644d8d51c874a4ae20a274: Status 404 returned error can't find the container with id 2b8e640b8cf79124d5304ffff22cf95b0bde0d8f88644d8d51c874a4ae20a274
Jul 01 15:04:40.031550 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.031523    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g6n5t_e4fa0250-95e6-4f97-9688-6c72a2910812/dns/0.log"
Jul 01 15:04:40.074103 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.074075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g6n5t_e4fa0250-95e6-4f97-9688-6c72a2910812/kube-rbac-proxy/0.log"
Jul 01 15:04:40.308502 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.308405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99" event={"ID":"bf4622f6-b68a-4f35-8f58-3c5d459103c2","Type":"ContainerStarted","Data":"1d506ba120967acafd244787e61aa304e09d9a6d50540aee99c969c9eca0b67a"}
Jul 01 15:04:40.308502 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.308477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99" event={"ID":"bf4622f6-b68a-4f35-8f58-3c5d459103c2","Type":"ContainerStarted","Data":"2b8e640b8cf79124d5304ffff22cf95b0bde0d8f88644d8d51c874a4ae20a274"}
Jul 01 15:04:40.308709 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.308556    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:40.319996 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.319969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9p5nd_35978eac-71cf-4911-a661-c713ab08588d/dns-node-resolver/0.log"
Jul 01 15:04:40.343957 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.343891    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99" podStartSLOduration=1.3438722969999999 podStartE2EDuration="1.343872297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 15:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 15:04:40.342629644 +0000 UTC m=+1558.868913649" watchObservedRunningTime="2026-07-01 15:04:40.343872297 +0000 UTC m=+1558.870156302"
Jul 01 15:04:40.941260 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:40.941225    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bwfq_e079a600-38ba-410b-949c-40bf6ef35e6f/node-ca/0.log"
Jul 01 15:04:42.029693 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:42.029663    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674f7cb5-nhjq7_b3ba0f3b-4fa8-43e1-8091-861ebf8eacdb/router/0.log"
Jul 01 15:04:42.559083 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:42.559052    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nfp67_6e3eb182-9ffa-4ca8-8b49-9c0683c23072/serve-healthcheck-canary/0.log"
Jul 01 15:04:42.991124 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:42.991091    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-l9tjh_c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255/insights-operator/1.log"
Jul 01 15:04:42.991584 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:42.991550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-l9tjh_c0e4ba3b-10c7-4a6a-9d75-de2d5bc04255/insights-operator/0.log"
Jul 01 15:04:43.237784 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:43.237759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpkp9_f3bec9de-cb72-4480-a078-60a9ab73b889/kube-rbac-proxy/0.log"
Jul 01 15:04:43.282916 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:43.282830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpkp9_f3bec9de-cb72-4480-a078-60a9ab73b889/exporter/0.log"
Jul 01 15:04:43.318915 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:43.318892    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpkp9_f3bec9de-cb72-4480-a078-60a9ab73b889/extractor/0.log"
Jul 01 15:04:45.753758 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:45.753727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-224zh_61d733f4-2e73-4994-9b20-49b0c9ab9dc2/s3-init/0.log"
Jul 01 15:04:45.773787 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:45.773763    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-224zh_61d733f4-2e73-4994-9b20-49b0c9ab9dc2/download-hf-model/0.log"
Jul 01 15:04:45.883881 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:45.883852    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7g9vd_ed8cd65b-5966-48f7-84a8-82391b9fae73/s3-tls-init-custom/0.log"
Jul 01 15:04:45.910550 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:45.910520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7g9vd_ed8cd65b-5966-48f7-84a8-82391b9fae73/download-hf-model/0.log"
Jul 01 15:04:45.946054 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:45.946025    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-fd7xr_d199f6c7-cdf9-46c4-9136-c64797996d94/s3-tls-init-serving/0.log"
Jul 01 15:04:45.965327 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:45.965284    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-fd7xr_d199f6c7-cdf9-46c4-9136-c64797996d94/download-hf-model/0.log"
Jul 01 15:04:46.332518 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:46.332490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cdwxh/perf-node-gather-daemonset-9hl99"
Jul 01 15:04:51.255739 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:51.255708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-8dfkr_d359dafa-4332-46a3-ae6c-4d6197a474f6/kube-storage-version-migrator-operator/1.log"
Jul 01 15:04:51.257791 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:51.257769    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-8dfkr_d359dafa-4332-46a3-ae6c-4d6197a474f6/kube-storage-version-migrator-operator/0.log"
Jul 01 15:04:52.445425 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.445394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/kube-multus-additional-cni-plugins/0.log"
Jul 01 15:04:52.473678 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.473651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/egress-router-binary-copy/0.log"
Jul 01 15:04:52.507458 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.507429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/cni-plugins/0.log"
Jul 01 15:04:52.540455 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.540424    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/bond-cni-plugin/0.log"
Jul 01 15:04:52.576624 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.576593    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/routeoverride-cni/0.log"
Jul 01 15:04:52.609369 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.609334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/whereabouts-cni-bincopy/0.log"
Jul 01 15:04:52.671006 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:52.670968    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bz8l6_3e44db6a-a5cd-430e-9f12-fb6ce04488ba/whereabouts-cni/0.log"
Jul 01 15:04:53.017976 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:53.017945    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-swpjb_6f9e4965-e925-4022-a963-8a03bc3b2c3c/kube-multus/0.log"
Jul 01 15:04:53.200591 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:53.200556    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vwzh5_12300cae-4145-449e-9bac-bf1b507ab3da/network-metrics-daemon/0.log"
Jul 01 15:04:53.225607 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:53.225566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vwzh5_12300cae-4145-449e-9bac-bf1b507ab3da/kube-rbac-proxy/0.log"
Jul 01 15:04:54.365739 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.365704    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-controller/0.log"
Jul 01 15:04:54.387545 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.387517    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/0.log"
Jul 01 15:04:54.387703 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.387554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovn-acl-logging/1.log"
Jul 01 15:04:54.409141 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.409112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/kube-rbac-proxy-node/0.log"
Jul 01 15:04:54.434206 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.434181    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 15:04:54.458661 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.458633    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/northd/0.log"
Jul 01 15:04:54.484671 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.484644    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/nbdb/0.log"
Jul 01 15:04:54.514528 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.514491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/sbdb/0.log"
Jul 01 15:04:54.563860 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:54.563829    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mpdnb_1d360fed-4183-4b87-b1b1-63d8c7964670/ovnkube-controller/0.log"
Jul 01 15:04:56.064461 ip-10-0-128-59 kubenswrapper[2579]: I0701 15:04:56.064430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ljxh4_dfe1e7e8-3db4-4a7b-b138-27044da30c8d/network-check-target-container/0.log"