Jul 08 16:31:58.416743 ip-10-0-137-118 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 16:31:58.416760 ip-10-0-137-118 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 16:31:58.416770 ip-10-0-137-118 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 16:31:58.417122 ip-10-0-137-118 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 16:32:08.436354 ip-10-0-137-118 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 08 16:32:08.436376 ip-10-0-137-118 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b1eb2bdbb5a740c2927829b6ca7be2d6 --
Jul 08 16:33:48.090521 ip-10-0-137-118 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.524856    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531182    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531202    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531206    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531209    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531212    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531215    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531218    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:48.610974 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531221    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:48.604748 ip-10-0-137-118 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531223    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531226    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531230    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531234    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531248    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531252    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531255    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531258    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531261    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531263    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531266    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531269    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531282    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531285    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531288    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531290    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531293    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531298    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:48.612716 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531302    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531305    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531308    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531311    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531314    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531317    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531320    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531323    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531327    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531330    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531333    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531336    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531338    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531341    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531344    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531346    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531349    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531352    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531354    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:48.614413 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531357    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531360    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531362    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531365    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531368    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531371    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531373    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531376    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531379    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531382    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531384    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531387    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531389    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531392    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531395    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531397    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531399    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531402    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531404    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531407    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:48.642679 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531410    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531412    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531415    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531418    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531420    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531423    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531426    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531428    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531430    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531433    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531435    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531438    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531440    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531442    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531445    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531447    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531450    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531452    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531455    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531457    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:48.645175 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531460    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531463    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531466    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531468    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531471    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531474    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531477    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531480    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531483    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531485    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531488    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531490    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531493    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.531496    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533678    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533690    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533693    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533696    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533699    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533703    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:48.654595 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533706    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533708    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533711    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533714    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533717    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533719    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533722    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533724    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533727    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533729    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533732    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533734    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533736    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533739    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533741    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533744    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533747    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533751    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533754    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:48.657198 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533758    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533761    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533763    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533767    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533770    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533773    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533776    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533779    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533782    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533785    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533787    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533790    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533793    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533795    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533798    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533800    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533803    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533806    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533808    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:48.824741 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533811    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533814    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533816    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533820    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533824    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533834    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533837    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533841    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533843    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533847    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533850    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533853    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533856    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533859    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533861    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533864    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533867    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533870    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533873    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533875    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:48.825513 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533881    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533884    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533887    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533889    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533892    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533895    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533897    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533900    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533902    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533905    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533907    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533910    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533913    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533917    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533920    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533923    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533925    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533927    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533930    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533934    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:48.826586 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533937    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533941    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533944    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533948    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533951    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533954    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533956    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533959    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533961    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533964    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533967    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533969    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533973    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533976    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.533978    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534358    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534363    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534366    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534369    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534372    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:48.827556 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534374    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534377    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534380    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534383    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534385    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534388    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534391    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534393    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534396    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534398    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534401    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534404    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534407    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534409    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534412    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534415    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534418    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534421    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534424    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:48.828160 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534426    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534429    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534431    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534434    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534437    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534441    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534443    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534446    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534449    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534452    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534456    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534460    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534463    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534468    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534470    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534473    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534475    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534479    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534481    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534484    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:48.829040 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534486    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534488    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534491    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534495    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534498    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534501    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534504    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534506    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534509    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534512    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534514    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534517    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534519    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534522    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534524    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534526    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534529    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534533    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534535    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:48.830751 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534538    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534540    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534543    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534545    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534548    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534551    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534556    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534558    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534560    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534564    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534568    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534570    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534573    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534576    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534579    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534582    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534585    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534588    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534591    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534595    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:48.832464 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534597    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534600    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534602    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534605    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534608    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534611    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534613    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534615    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534618    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534620    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534624    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534626    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534629    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534632    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534634    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.534637    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534706    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534713    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534723    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534727    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:33:48.833122 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534730    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534734    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534737    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534741    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534744    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534747    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534751    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534754    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534757    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534760    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534763    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534766    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534771    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534774    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534778    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534781    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534784    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534787    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534790    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534794    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534796    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534799    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534804    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534807    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:33:48.835510 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534810    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534813    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534816    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534819    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534823    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534826    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534830    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534833    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534836    2559 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534839    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534843    2559 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534846    2559 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534849    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534852    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534855    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534858    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534861    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534864    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534867    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534869    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534872    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534876    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534879    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534881    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534884    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:33:48.837646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534887    2559 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534891    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534894    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534896    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534899    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534903    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534906    2559 flags.go:64] FLAG: --help="false"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534909    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534912    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534915    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534918    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534921    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534925    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534929    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534932    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534935    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534938    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534940    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534943    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534946    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534949    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534952    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534955    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534957    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:33:48.839770 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534960    2559 flags.go:64] FLAG: --lock-file=""
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534963    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534966    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534969    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534974    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534977    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534980    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534983    2559 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534985    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534989    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534991    2559 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534994    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.534998    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535003    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535006    2559 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535009    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535012    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535020    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535023    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535026    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535029    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535033    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535040    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535043    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535046    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:33:48.845739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535049    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535052    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535057    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535060    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535063    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535066    2559 flags.go:64] FLAG: --port="10250"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535069    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535072    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0abfb1b49b0de9944"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535075    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535078    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535081    2559 flags.go:64] FLAG: --register-node="true"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535084    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535087    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535090    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535093    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535096    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535099    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535103    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535106    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535108    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535111    2559 flags.go:64] FLAG: --runonce="false"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535115    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535118    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535121    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535124    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:33:48.846504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535126    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535129    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535132    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535136    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535139    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535142    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535144    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535147    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535150    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535153    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535156    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535161    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535163    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535166    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535171    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535174    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535177    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535180    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535182    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535185    2559 flags.go:64] FLAG: --v="2"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535190    2559 flags.go:64] FLAG: --version="false"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535194    2559 flags.go:64] FLAG: --vmodule=""
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535197    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.535200    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535330    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:48.850170 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535334    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535337    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535340    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535345    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535348    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535350    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535353    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535356    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535359    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535361    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535366    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535369    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535371    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535374    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535377    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535379    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535382    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535384    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535387    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:48.850876 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535389    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535392    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535395    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535397    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535400    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535402    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535405    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535408    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535410    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535413    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535416    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535419    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535421    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535424    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535427    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535429    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535433    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535435    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535438    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535440    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:48.851390 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535443    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535445    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535448    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535451    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535454    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535457    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535459    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535462    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535464    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535467    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535469    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535472    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535474    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535479    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535482    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535485    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535488    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535490    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535493    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:48.851949 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535495    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535498    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535500    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535503    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535506    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535508    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535511    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535513    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535516    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535519    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535522    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535524    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535527    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535529    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535533    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535536    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535540    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535543    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535546    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:48.852492 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535549    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535551    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535554    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535557    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535559    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535562    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535565    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535568    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535572    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535575    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535578    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535581    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535583    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535586    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535588    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535591    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535593    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535596    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535599    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535602    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:49.093829 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.535604    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.536380    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.545172    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.545190    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.545208    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.545214    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545284    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545289    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545293    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545296    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545299    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545303    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545305    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545308    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545311    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:49.094951 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545313    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545316    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545319    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545321    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545324    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545326    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545329    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545332    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545334    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545337    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545339    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545342    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545344    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545347    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545350    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545354    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545357    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545360    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545362    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545365    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:49.095549 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545368    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545371    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545374    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545376    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545379    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545382    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545384    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545387    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545389    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545392    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545395    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545397    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545400    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545402    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545405    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545408    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545410    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545413    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545416    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545419    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:49.096144 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545422    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545424    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545427    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545430    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545432    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545437    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545440    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545443    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545445    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545448    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545451    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545454    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545456    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545459    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545462    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545472    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545475    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545479    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545483    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545487    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:49.096729 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545490    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545493    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545495    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545498    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545501    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545504    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545507    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545509    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545512    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545515    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545518    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545521    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545524    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545527    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545530    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545532    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545535    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545539    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545541    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545544    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:49.097280 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545546    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545550    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545553    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545556    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545559    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545562    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545566    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545569    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545573    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545576    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.545581    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545686    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545690    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545693    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:49.097859 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545696    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545699    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545703    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545706    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545708    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545711    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545714    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545716    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545719    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545722    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545724    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545727    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545730    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545732    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545735    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545737    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545742    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545744    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545747    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545750    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:49.098271 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545752    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545755    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545758    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545760    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545763    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545766    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545768    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545771    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545774    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545778    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545780    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545783    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545786    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545789    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545792    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545795    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545797    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545800    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545803    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:49.098832 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545805    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545808    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545812    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545814    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545817    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545820    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545822    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545825    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545827    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545831    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545834    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545836    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545839    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545841    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545844    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545847    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545849    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545851    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545854    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545861    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:49.099349 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545863    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545866    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545868    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545871    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545873    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545876    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545878    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545881    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545883    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545886    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545889    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545891    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545894    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545896    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545900    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545903    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545905    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545908    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545910    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545913    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:49.099891 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545915    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545919    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545923    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545926    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545929    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545932    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545935    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545938    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545940    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545943    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545947    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545949    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545952    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545955    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545957    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545960    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:49.100426 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:48.545962    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.545968    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.546060    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.548464    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.550222    2559 server.go:1019] "Starting client certificate rotation"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.550330    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.550368    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.553006    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.554377    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.566848    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.569339    2559 log.go:25] "Validated CRI v1 image API"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.570112    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:33:49.100891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.573615    2559 fs.go:135] Filesystem UUIDs: map[5000cc87-f9eb-4578-a4db-14aa83e9bbcc:/dev/nvme0n1p4 6bb7432f-cf3d-4136-a5e9-e130e02e1c25:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 08 16:33:49.101231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.573662    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.580308    2559 manager.go:216] Machine: {Timestamp:2026-07-08 16:33:48.578913563 +0000 UTC m=+0.383767413 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095533 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec299c5199e46e9877f1797cd261c50a SystemUUID:ec299c51-99e4-6e98-77f1-797cd261c50a BootID:b1eb2bdb-b5a7-40c2-9278-29b6ca7be2d6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:64:19:ba:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:64:19:ba:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:7a:2f:04:07:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.580407    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.580486    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.583900    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.583925    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.584770    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.584780    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:33:49.101344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.584788    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.585085    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.585749    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.588493    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593281    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593397    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593425    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593434    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593449    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593457    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593475    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593824    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.593839    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.596229    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.598651    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.598678    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600010    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600023    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600029    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600035    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600044    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600053    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600066    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600073    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600080    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600089    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600103    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600109    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600921    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.600930    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.603965    2559 server.go:1268] "Started kubelet"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.604060    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.604088    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.604131    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.605525    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.606553    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.610732    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613144    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613151    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.613223    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613884    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613898    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613883    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613984    2559 factory.go:55] Registering systemd factory
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613993    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:33:49.101908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.613999    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614001    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614188    2559 factory.go:153] Registering CRI-O factory
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614201    2559 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614285    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614304    2559 factory.go:103] Registering Raw factory
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614315    2559 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.614739    2559 manager.go:349] Starting recovery of all containers
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.619182    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.619399    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.619463    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.619574    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.619641    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da801bf9e25  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.603940389 +0000 UTC m=+0.408794237,LastTimestamp:2026-07-08 16:33:48.603940389 +0000 UTC m=+0.408794237,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.624152    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.630032    2559 manager.go:354] Recovery completed
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.634586    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636391    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636432    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636442    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636827    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636827    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636848    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.636863    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.637393    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.637412    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.637421    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.638967    2559 policy_none.go:49] "None policy: Start"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.638982    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:33:49.103133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.638996    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.639395    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af3305  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636418821 +0000 UTC m=+0.441272669,LastTimestamp:2026-07-08 16:33:48.636418821 +0000 UTC m=+0.441272669,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.641032    2559 policy_none.go:47] "Start"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.651440    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af7852  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636436562 +0000 UTC m=+0.441290411,LastTimestamp:2026-07-08 16:33:48.636436562 +0000 UTC m=+0.441290411,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.661014    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af9d81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636446081 +0000 UTC m=+0.441299930,LastTimestamp:2026-07-08 16:33:48.636446081 +0000 UTC m=+0.441299930,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.666758    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9qrl"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.672125    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-118.ec2.internal.18c05da803af3305\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af3305  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636418821 +0000 UTC m=+0.441272669,LastTimestamp:2026-07-08 16:33:48.637401786 +0000 UTC m=+0.442255635,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.360685 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.674804    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g9qrl"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.682618    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-118.ec2.internal.18c05da803af7852\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af7852  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636436562 +0000 UTC m=+0.441290411,LastTimestamp:2026-07-08 16:33:48.63741657 +0000 UTC m=+0.442270421,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.690395    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-118.ec2.internal.18c05da803af9d81\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af9d81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636446081 +0000 UTC m=+0.441299930,LastTimestamp:2026-07-08 16:33:48.637424658 +0000 UTC m=+0.442278510,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.693767    2559 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.693795    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.693806    2559 server.go:87] "Starting device plugin registration server"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.694021    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.694036    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.694111    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.694204    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.694214    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.694560    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.694598    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.711199    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da80741a196  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.69634703 +0000 UTC m=+0.501200866,LastTimestamp:2026-07-08 16:33:48.69634703 +0000 UTC m=+0.501200866,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.751021    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.752122    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.752156    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.752176    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:33:49.361659 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.752210    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.762036    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.795116    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.795842    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.795866    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.795876    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.795895    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.804863    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-118.ec2.internal.18c05da803af3305\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af3305  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636418821 +0000 UTC m=+0.441272669,LastTimestamp:2026-07-08 16:33:48.795852047 +0000 UTC m=+0.600705896,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.815145    2559 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.815221    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-118.ec2.internal.18c05da803af7852\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-118.ec2.internal.18c05da803af7852  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-118.ec2.internal,UID:ip-10-0-137-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-118.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-118.ec2.internal,},FirstTimestamp:2026-07-08 16:33:48.636436562 +0000 UTC m=+0.441290411,LastTimestamp:2026-07-08 16:33:48.7958709 +0000 UTC m=+0.600724750,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-118.ec2.internal,}"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.831910    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-118.ec2.internal\" not found" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.853264    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"]
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.853317    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.854043    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.854067    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.854078    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.856486    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.856596    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.856618    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.857121    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.857135    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.857156    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.857168    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.857142    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.857233    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.859479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.859500    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.861648    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.861672    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.861681    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.883797    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-118.ec2.internal\" not found" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:48.889010    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-118.ec2.internal\" not found" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.915429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc853c8f2f1ea6a968532ff9703fcfa8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal\" (UID: \"cc853c8f2f1ea6a968532ff9703fcfa8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.915450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc853c8f2f1ea6a968532ff9703fcfa8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal\" (UID: \"cc853c8f2f1ea6a968532ff9703fcfa8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:48.915464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e23d70a312ac711b759fd35bb366a792-config\") pod \"kube-apiserver-proxy-ip-10-0-137-118.ec2.internal\" (UID: \"e23d70a312ac711b759fd35bb366a792\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015255    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e23d70a312ac711b759fd35bb366a792-config\") pod \"kube-apiserver-proxy-ip-10-0-137-118.ec2.internal\" (UID: \"e23d70a312ac711b759fd35bb366a792\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc853c8f2f1ea6a968532ff9703fcfa8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal\" (UID: \"cc853c8f2f1ea6a968532ff9703fcfa8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc853c8f2f1ea6a968532ff9703fcfa8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal\" (UID: \"cc853c8f2f1ea6a968532ff9703fcfa8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.362749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015642    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc853c8f2f1ea6a968532ff9703fcfa8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal\" (UID: \"cc853c8f2f1ea6a968532ff9703fcfa8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015680    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e23d70a312ac711b759fd35bb366a792-config\") pod \"kube-apiserver-proxy-ip-10-0-137-118.ec2.internal\" (UID: \"e23d70a312ac711b759fd35bb366a792\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.015702    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cc853c8f2f1ea6a968532ff9703fcfa8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal\" (UID: \"cc853c8f2f1ea6a968532ff9703fcfa8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.016089    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.016105    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.016114    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.016132    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.027088    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:49.027102    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-118.ec2.internal\": node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:49.064581    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:49.164947    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.188084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:49.191543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:49.363341 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:49.265733    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:51.484320 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:49.366226    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:51.484320 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:49.466697    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-118.ec2.internal\" not found"
Jul 08 16:33:51.484320 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.442871    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:51.484320 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.464371    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:33:51.484320 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.464471    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.509128    2559 apiserver.go:52] "Watching apiserver"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.523300    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.524255    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kn2hm","openshift-image-registry/node-ca-sjfpx","openshift-multus/multus-additional-cni-plugins-b45pc","openshift-multus/multus-gz6lt","openshift-network-diagnostics/network-check-target-mlbdg","openshift-network-operator/iptables-alerter-szs64","kube-system/konnectivity-agent-bdrjd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4","openshift-cluster-node-tuning-operator/tuned-4x75t","openshift-dns/node-resolver-lhssb","openshift-multus/network-metrics-daemon-txdt2","openshift-ovn-kubernetes/ovnkube-node-xbnrp"]
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.526725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.528651    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.528651    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.528709    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.528799    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rml4q\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.528851    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.528900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.529265    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.530773    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.530792    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.530976    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kzhr4\""
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.531088    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.531151    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533213    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533802    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61812833-1331-488b-96bd-e32d6fc6323e-host-slash\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533822    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/946032cc-95d9-4a74-ab85-10e9c5bc9051-hosts-file\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946032cc-95d9-4a74-ab85-10e9c5bc9051-tmp-dir\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck86s\" (UniqueName: \"kubernetes.io/projected/61812833-1331-488b-96bd-e32d6fc6323e-kube-api-access-ck86s\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.718885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95bpg\" (UniqueName: \"kubernetes.io/projected/946032cc-95d9-4a74-ab85-10e9c5bc9051-kube-api-access-95bpg\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/598f3885-6ab6-4276-a3f0-8b6e634cf091-kubelet-config\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/598f3885-6ab6-4276-a3f0-8b6e634cf091-dbus\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.533970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.534001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61812833-1331-488b-96bd-e32d6fc6323e-iptables-alerter-script\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.534796    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.534831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.535107    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.535224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nq8p7\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.535425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.539198    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.539660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.539711    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.541276    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.541512    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.541525    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.541673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.541714    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.541718    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.543833    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.545349    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.545363    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w79xx\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.545638    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.545843    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:33:51.720166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546139    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ldstl\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546162    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546189    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2qgv7\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-htztx\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546505    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.546541    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.548610    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.550315    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpkml\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.550611    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.550612    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.550885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.550928    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.553103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.555529    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.555664    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.557215    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.557265    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.557268    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5p56q\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.557339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.557491    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.563643    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xd94s"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.573265    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal"]
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.573491    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.573535    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.574512    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xd94s"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.583546    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal"]
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.583756    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.630371    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:33:51.721231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-var-lib-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-cni-netd\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d19257c2-bde8-4433-bfdc-71e91f1c3caa-serviceca\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6v9\" (UniqueName: \"kubernetes.io/projected/fbbf911d-083c-4b51-a7b4-b795abbc5e70-kube-api-access-wq6v9\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-kubernetes\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck86s\" (UniqueName: \"kubernetes.io/projected/61812833-1331-488b-96bd-e32d6fc6323e-kube-api-access-ck86s\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95bpg\" (UniqueName: \"kubernetes.io/projected/946032cc-95d9-4a74-ab85-10e9c5bc9051-kube-api-access-95bpg\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634804    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-node-log\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634819    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-cni-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-cnibin\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.634903    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:51.722469 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.634958    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:52.134935852 +0000 UTC m=+2.024350839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.634981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-etc-selinux\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-sys\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61812833-1331-488b-96bd-e32d6fc6323e-iptables-alerter-script\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/946032cc-95d9-4a74-ab85-10e9c5bc9051-hosts-file\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-systemd-units\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-systemd\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635111    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/946032cc-95d9-4a74-ab85-10e9c5bc9051-hosts-file\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-daemon-config\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635148    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cnibin\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/598f3885-6ab6-4276-a3f0-8b6e634cf091-kubelet-config\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cni-binary-copy\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635195    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-lib-modules\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.723072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61812833-1331-488b-96bd-e32d6fc6323e-host-slash\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635222    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/598f3885-6ab6-4276-a3f0-8b6e634cf091-kubelet-config\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-etc-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635251    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61812833-1331-488b-96bd-e32d6fc6323e-host-slash\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-system-cni-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjg7\" (UniqueName: \"kubernetes.io/projected/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-kube-api-access-lgjg7\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-hostroot\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90db286d-0405-4190-8aa9-42695ef44a41-agent-certs\") pod \"konnectivity-agent-bdrjd\" (UID: \"90db286d-0405-4190-8aa9-42695ef44a41\") " pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-modprobe-d\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysctl-d\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-run\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946032cc-95d9-4a74-ab85-10e9c5bc9051-tmp-dir\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-run-netns\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.723645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vc5r\" (UniqueName: \"kubernetes.io/projected/d19257c2-bde8-4433-bfdc-71e91f1c3caa-kube-api-access-7vc5r\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-socket-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635597    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-tmp\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/598f3885-6ab6-4276-a3f0-8b6e634cf091-dbus\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-ovn\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovnkube-config\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-cni-binary-copy\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovn-node-metrics-cert\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90db286d-0405-4190-8aa9-42695ef44a41-konnectivity-ca\") pod \"konnectivity-agent-bdrjd\" (UID: \"90db286d-0405-4190-8aa9-42695ef44a41\") " pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635785    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/598f3885-6ab6-4276-a3f0-8b6e634cf091-dbus\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635804    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-socket-dir-parent\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-netns\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-conf-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-kubelet\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovnkube-script-lib\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-var-lib-kubelet\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzq8v\" (UniqueName: \"kubernetes.io/projected/33c52f87-ae67-40cb-8711-1a37de967be6-kube-api-access-nzq8v\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.635985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-etc-kubernetes\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-tuned\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-k8s-cni-cncf-io\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-kubelet\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-registration-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-device-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysconfig\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636112    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2rp\" (UniqueName: \"kubernetes.io/projected/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-kube-api-access-hf2rp\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.724769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-env-overrides\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-cni-multus\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-system-cni-dir\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-os-release\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636201    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61812833-1331-488b-96bd-e32d6fc6323e-iptables-alerter-script\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysctl-conf\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636241    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-cni-bin\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-slash\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57njz\" (UniqueName: \"kubernetes.io/projected/6b63cc5a-ba78-4672-b37a-d49c28673f50-kube-api-access-57njz\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d19257c2-bde8-4433-bfdc-71e91f1c3caa-host\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-cni-bin\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.725286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-sys-fs\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2g6\" (UniqueName: \"kubernetes.io/projected/10de1d9b-34d5-45e6-8809-232627883766-kube-api-access-8t2g6\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-multus-certs\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-log-socket\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-os-release\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-systemd\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-host\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.636896    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946032cc-95d9-4a74-ab85-10e9c5bc9051-tmp-dir\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.644406    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.646074    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.649030    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95bpg\" (UniqueName: \"kubernetes.io/projected/946032cc-95d9-4a74-ab85-10e9c5bc9051-kube-api-access-95bpg\") pod \"node-resolver-lhssb\" (UID: \"946032cc-95d9-4a74-ab85-10e9c5bc9051\") " pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.725816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.649037    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck86s\" (UniqueName: \"kubernetes.io/projected/61812833-1331-488b-96bd-e32d6fc6323e-kube-api-access-ck86s\") pod \"iptables-alerter-szs64\" (UID: \"61812833-1331-488b-96bd-e32d6fc6323e\") " pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.729035    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:28:50 +0000 UTC" deadline="2028-01-30 21:33:03.009734014 +0000 UTC"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.729074    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13708h59m11.280662334s"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovn-node-metrics-cert\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90db286d-0405-4190-8aa9-42695ef44a41-konnectivity-ca\") pod \"konnectivity-agent-bdrjd\" (UID: \"90db286d-0405-4190-8aa9-42695ef44a41\") " pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-socket-dir-parent\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-netns\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-conf-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-kubelet\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovnkube-script-lib\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736866    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-netns\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736929    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-conf-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-var-lib-kubelet\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736973    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-var-lib-kubelet\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.932844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736866    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-socket-dir-parent\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzq8v\" (UniqueName: \"kubernetes.io/projected/33c52f87-ae67-40cb-8711-1a37de967be6-kube-api-access-nzq8v\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.736932    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-kubelet\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-etc-kubernetes\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-tuned\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-k8s-cni-cncf-io\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-kubelet\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-registration-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-device-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysconfig\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2rp\" (UniqueName: \"kubernetes.io/projected/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-kube-api-access-hf2rp\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-env-overrides\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-cni-multus\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.933499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737358    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90db286d-0405-4190-8aa9-42695ef44a41-konnectivity-ca\") pod \"konnectivity-agent-bdrjd\" (UID: \"90db286d-0405-4190-8aa9-42695ef44a41\") " pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-system-cni-dir\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737400    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-system-cni-dir\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-os-release\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737423    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-etc-kubernetes\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737433    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovnkube-script-lib\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysctl-conf\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737443    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-k8s-cni-cncf-io\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737378    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-cni-bin\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737508    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-kubelet\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737520    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-os-release\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-slash\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.934072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737542    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-registration-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57njz\" (UniqueName: \"kubernetes.io/projected/6b63cc5a-ba78-4672-b37a-d49c28673f50-kube-api-access-57njz\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737552    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysctl-conf\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737558    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-device-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737584    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-cni-multus\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737587    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d19257c2-bde8-4433-bfdc-71e91f1c3caa-host\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737603    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-cni-bin\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737482    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysconfig\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-cni-bin\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-sys-fs\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737658    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d19257c2-bde8-4433-bfdc-71e91f1c3caa-host\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.737667    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2g6\" (UniqueName: \"kubernetes.io/projected/10de1d9b-34d5-45e6-8809-232627883766-kube-api-access-8t2g6\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737712    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-slash\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.934519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-multus-certs\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.737751    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:52.237732092 +0000 UTC m=+2.127147080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737758    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-sys-fs\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737762    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-run-multus-certs\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737793    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-host-var-lib-cni-bin\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-log-socket\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-os-release\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737829    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-run-ovn-kubernetes\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-systemd\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737868    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-log-socket\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-host\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737894    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-env-overrides\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-var-lib-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737910    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-os-release\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737917    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-host\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-cni-netd\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737932    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-systemd\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737956    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-var-lib-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.737985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d19257c2-bde8-4433-bfdc-71e91f1c3caa-serviceca\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738021    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-cni-netd\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6v9\" (UniqueName: \"kubernetes.io/projected/fbbf911d-083c-4b51-a7b4-b795abbc5e70-kube-api-access-wq6v9\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-kubernetes\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-node-log\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-cni-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-cnibin\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-etc-selinux\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-sys\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.935620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-systemd-units\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738307    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-node-log\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-systemd\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738335    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-cni-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738366    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-cnibin\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-daemon-config\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738407    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-systemd\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738411    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d19257c2-bde8-4433-bfdc-71e91f1c3caa-serviceca\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cnibin\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738438    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cnibin\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738449    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-systemd-units\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738245    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-kubernetes\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cni-binary-copy\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-lib-modules\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.936245 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738502    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-etc-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738548    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-etc-openvswitch\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738561    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-etc-selinux\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-system-cni-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjg7\" (UniqueName: \"kubernetes.io/projected/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-kube-api-access-lgjg7\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-hostroot\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738628    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-lib-modules\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738673    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90db286d-0405-4190-8aa9-42695ef44a41-agent-certs\") pod \"konnectivity-agent-bdrjd\" (UID: \"90db286d-0405-4190-8aa9-42695ef44a41\") " pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738705    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-system-cni-dir\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-modprobe-d\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysctl-d\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738747    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-hostroot\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-run\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-run-netns\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.936785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738819    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-modprobe-d\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vc5r\" (UniqueName: \"kubernetes.io/projected/d19257c2-bde8-4433-bfdc-71e91f1c3caa-kube-api-access-7vc5r\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738883    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-run\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-socket-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738935    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cni-binary-copy\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-tmp\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738981    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-sysctl-d\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.738990    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-run-netns\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-ovn\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovnkube-config\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739035    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-multus-daemon-config\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.937379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-cni-binary-copy\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739068    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739129    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10de1d9b-34d5-45e6-8809-232627883766-socket-dir\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739023    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739311    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b63cc5a-ba78-4672-b37a-d49c28673f50-run-ovn\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739446    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbbf911d-083c-4b51-a7b4-b795abbc5e70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739448    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbbf911d-083c-4b51-a7b4-b795abbc5e70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739520    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-sys\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739553    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovnkube-config\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.739533    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-cni-binary-copy\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.740079    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-etc-tuned\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.740945    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b63cc5a-ba78-4672-b37a-d49c28673f50-ovn-node-metrics-cert\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.741309    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-tmp\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.741339    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90db286d-0405-4190-8aa9-42695ef44a41-agent-certs\") pod \"konnectivity-agent-bdrjd\" (UID: \"90db286d-0405-4190-8aa9-42695ef44a41\") " pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.751668    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:51.937849 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.751710    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.751725    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.751792    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:52.251773909 +0000 UTC m=+2.141188916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.753171    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vc5r\" (UniqueName: \"kubernetes.io/projected/d19257c2-bde8-4433-bfdc-71e91f1c3caa-kube-api-access-7vc5r\") pod \"node-ca-sjfpx\" (UID: \"d19257c2-bde8-4433-bfdc-71e91f1c3caa\") " pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.754030    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2g6\" (UniqueName: \"kubernetes.io/projected/10de1d9b-34d5-45e6-8809-232627883766-kube-api-access-8t2g6\") pod \"aws-ebs-csi-driver-node-ksgm4\" (UID: \"10de1d9b-34d5-45e6-8809-232627883766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.754769    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6v9\" (UniqueName: \"kubernetes.io/projected/fbbf911d-083c-4b51-a7b4-b795abbc5e70-kube-api-access-wq6v9\") pod \"multus-additional-cni-plugins-b45pc\" (UID: \"fbbf911d-083c-4b51-a7b4-b795abbc5e70\") " pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.754998    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57njz\" (UniqueName: \"kubernetes.io/projected/6b63cc5a-ba78-4672-b37a-d49c28673f50-kube-api-access-57njz\") pod \"ovnkube-node-xbnrp\" (UID: \"6b63cc5a-ba78-4672-b37a-d49c28673f50\") " pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.755001    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzq8v\" (UniqueName: \"kubernetes.io/projected/33c52f87-ae67-40cb-8711-1a37de967be6-kube-api-access-nzq8v\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.758179    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjg7\" (UniqueName: \"kubernetes.io/projected/dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa-kube-api-access-lgjg7\") pod \"multus-gz6lt\" (UID: \"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa\") " pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.758330    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2rp\" (UniqueName: \"kubernetes.io/projected/c29ba8ac-fe57-4260-9bee-92c4758bcbe5-kube-api-access-hf2rp\") pod \"tuned-4x75t\" (UID: \"c29ba8ac-fe57-4260-9bee-92c4758bcbe5\") " pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.855064    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.862705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-szs64"
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.866137    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc853c8f2f1ea6a968532ff9703fcfa8.slice/crio-61baee2bfabe41c2b9086eb65b5f3158f439a94538ce0b6f439d34812310739e WatchSource:0}: Error finding container 61baee2bfabe41c2b9086eb65b5f3158f439a94538ce0b6f439d34812310739e: Status 404 returned error can't find the container with id 61baee2bfabe41c2b9086eb65b5f3158f439a94538ce0b6f439d34812310739e
Jul 08 16:33:51.938396 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.866788    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23d70a312ac711b759fd35bb366a792.slice/crio-cc90cd01380cdf1f013729b75352649597e6a75cbba7bdb27bcf9ba0235d01a8 WatchSource:0}: Error finding container cc90cd01380cdf1f013729b75352649597e6a75cbba7bdb27bcf9ba0235d01a8: Status 404 returned error can't find the container with id cc90cd01380cdf1f013729b75352649597e6a75cbba7bdb27bcf9ba0235d01a8
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.871907    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61812833_1331_488b_96bd_e32d6fc6323e.slice/crio-708eec28c631fba39b445f1928b9437bd38a0745f77751cd53fbfa551e245231 WatchSource:0}: Error finding container 708eec28c631fba39b445f1928b9437bd38a0745f77751cd53fbfa551e245231: Status 404 returned error can't find the container with id 708eec28c631fba39b445f1928b9437bd38a0745f77751cd53fbfa551e245231
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.871933    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.882518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lhssb"
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.892931    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946032cc_95d9_4a74_ab85_10e9c5bc9051.slice/crio-feef5fd9f477cc4a8ac87688e6a9ae88ad790dcec99e5e4ce9e505bdcc54ae98 WatchSource:0}: Error finding container feef5fd9f477cc4a8ac87688e6a9ae88ad790dcec99e5e4ce9e505bdcc54ae98: Status 404 returned error can't find the container with id feef5fd9f477cc4a8ac87688e6a9ae88ad790dcec99e5e4ce9e505bdcc54ae98
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.901822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sjfpx"
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.912037    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19257c2_bde8_4433_bfdc_71e91f1c3caa.slice/crio-def95951af6fbb698ed97c809bfca5c66a30368959b0eef72a2be9af376663ce WatchSource:0}: Error finding container def95951af6fbb698ed97c809bfca5c66a30368959b0eef72a2be9af376663ce: Status 404 returned error can't find the container with id def95951af6fbb698ed97c809bfca5c66a30368959b0eef72a2be9af376663ce
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.917375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b45pc"
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.924839    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbf911d_083c_4b51_a7b4_b795abbc5e70.slice/crio-9048f0f4bb7300969f4f0e368305f3dc7e74912561e2c5fa36ba8359bb5f5ca3 WatchSource:0}: Error finding container 9048f0f4bb7300969f4f0e368305f3dc7e74912561e2c5fa36ba8359bb5f5ca3: Status 404 returned error can't find the container with id 9048f0f4bb7300969f4f0e368305f3dc7e74912561e2c5fa36ba8359bb5f5ca3
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.927585    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gz6lt"
Jul 08 16:33:51.938863 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.935627    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfc731a_fc64_46f4_bdc7_ebb83b39d5aa.slice/crio-a6a3e1b35b5f36f19bebc57799fe65eba9862944c1fc04b58170a2cf4103aab3 WatchSource:0}: Error finding container a6a3e1b35b5f36f19bebc57799fe65eba9862944c1fc04b58170a2cf4103aab3: Status 404 returned error can't find the container with id a6a3e1b35b5f36f19bebc57799fe65eba9862944c1fc04b58170a2cf4103aab3
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.943898    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.950661    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90db286d_0405_4190_8aa9_42695ef44a41.slice/crio-a6764d730628a6b774486be01372865d9820c662d7422ac2ce8bb7ef524af7b0 WatchSource:0}: Error finding container a6764d730628a6b774486be01372865d9820c662d7422ac2ce8bb7ef524af7b0: Status 404 returned error can't find the container with id a6764d730628a6b774486be01372865d9820c662d7422ac2ce8bb7ef524af7b0
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.959170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4"
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.964769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4x75t"
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.966298    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10de1d9b_34d5_45e6_8809_232627883766.slice/crio-de5b787b40a0ee93ffb568b44a3e7d7a928066389f671eed3879f94571ccbd6d WatchSource:0}: Error finding container de5b787b40a0ee93ffb568b44a3e7d7a928066389f671eed3879f94571ccbd6d: Status 404 returned error can't find the container with id de5b787b40a0ee93ffb568b44a3e7d7a928066389f671eed3879f94571ccbd6d
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.972286    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29ba8ac_fe57_4260_9bee_92c4758bcbe5.slice/crio-c14851112938fa1e1dc5a00b3d98ea5527d31cd705123f086034ac3761d71b20 WatchSource:0}: Error finding container c14851112938fa1e1dc5a00b3d98ea5527d31cd705123f086034ac3761d71b20: Status 404 returned error can't find the container with id c14851112938fa1e1dc5a00b3d98ea5527d31cd705123f086034ac3761d71b20
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:51.974233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:33:51.982612    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b63cc5a_ba78_4672_b37a_d49c28673f50.slice/crio-e127d5f917089b42f286348d45caa230d5cf2aeb13778069c1f42aff7af9cd11 WatchSource:0}: Error finding container e127d5f917089b42f286348d45caa230d5cf2aeb13778069c1f42aff7af9cd11: Status 404 returned error can't find the container with id e127d5f917089b42f286348d45caa230d5cf2aeb13778069c1f42aff7af9cd11
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.984015    2559 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-xbnrp_openshift-ovn-kubernetes(6b63cc5a-ba78-4672-b37a-d49c28673f50): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.985180    2559 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-xbnrp_openshift-ovn-kubernetes(6b63cc5a-ba78-4672-b37a-d49c28673f50): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 08 16:33:52.157533 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:51.987932    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d26498fbce9f8ab56799e6e6f1920bc238d2d31f38945eab91ebc5f68b43dc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" podUID="6b63cc5a-ba78-4672-b37a-d49c28673f50"
Jul 08 16:33:52.161189 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.143074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:52.161189 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.143197    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:52.161189 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.143234    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:53.143222746 +0000 UTC m=+3.032637734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.244227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.244339    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.244381    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:53.244369697 +0000 UTC m=+3.133784688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.270638    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.345176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.345312    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.345337    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.345347    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:52.354568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.345397    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:53.345383756 +0000 UTC m=+3.234798745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:52.588858 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.577709    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:28:51 +0000 UTC" deadline="2028-03-03 17:44:27.337704207 +0000 UTC"
Jul 08 16:33:52.588858 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.577767    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14497h10m34.759941297s"
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.682018    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.682143    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.682512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.682600    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.686308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4" event={"ID":"10de1d9b-34d5-45e6-8809-232627883766","Type":"ContainerStarted","Data":"de5b787b40a0ee93ffb568b44a3e7d7a928066389f671eed3879f94571ccbd6d"}
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.727197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhssb" event={"ID":"946032cc-95d9-4a74-ab85-10e9c5bc9051","Type":"ContainerStarted","Data":"feef5fd9f477cc4a8ac87688e6a9ae88ad790dcec99e5e4ce9e505bdcc54ae98"}
Jul 08 16:33:52.738853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.732956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-szs64" event={"ID":"61812833-1331-488b-96bd-e32d6fc6323e","Type":"ContainerStarted","Data":"708eec28c631fba39b445f1928b9437bd38a0745f77751cd53fbfa551e245231"}
Jul 08 16:33:52.760209 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.755985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal" event={"ID":"cc853c8f2f1ea6a968532ff9703fcfa8","Type":"ContainerStarted","Data":"61baee2bfabe41c2b9086eb65b5f3158f439a94538ce0b6f439d34812310739e"}
Jul 08 16:33:52.760209 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.758106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdrjd" event={"ID":"90db286d-0405-4190-8aa9-42695ef44a41","Type":"ContainerStarted","Data":"a6764d730628a6b774486be01372865d9820c662d7422ac2ce8bb7ef524af7b0"}
Jul 08 16:33:52.765132 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.763412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gz6lt" event={"ID":"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa","Type":"ContainerStarted","Data":"a6a3e1b35b5f36f19bebc57799fe65eba9862944c1fc04b58170a2cf4103aab3"}
Jul 08 16:33:52.770524 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.768788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerStarted","Data":"9048f0f4bb7300969f4f0e368305f3dc7e74912561e2c5fa36ba8359bb5f5ca3"}
Jul 08 16:33:52.774488 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.773904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sjfpx" event={"ID":"d19257c2-bde8-4433-bfdc-71e91f1c3caa","Type":"ContainerStarted","Data":"def95951af6fbb698ed97c809bfca5c66a30368959b0eef72a2be9af376663ce"}
Jul 08 16:33:52.830790 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.777500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal" event={"ID":"e23d70a312ac711b759fd35bb366a792","Type":"ContainerStarted","Data":"cc90cd01380cdf1f013729b75352649597e6a75cbba7bdb27bcf9ba0235d01a8"}
Jul 08 16:33:52.830790 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.779878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"e127d5f917089b42f286348d45caa230d5cf2aeb13778069c1f42aff7af9cd11"}
Jul 08 16:33:52.830790 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:52.785652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4x75t" event={"ID":"c29ba8ac-fe57-4260-9bee-92c4758bcbe5","Type":"ContainerStarted","Data":"c14851112938fa1e1dc5a00b3d98ea5527d31cd705123f086034ac3761d71b20"}
Jul 08 16:33:52.830790 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:52.802795    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d26498fbce9f8ab56799e6e6f1920bc238d2d31f38945eab91ebc5f68b43dc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52d26498fbce9f8ab56799e6e6f1920bc238d2d31f38945eab91ebc5f68b43dc\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72c58cd87dea47bcff20bc07f2a9a5def3716e1a5b3d3e46ddcca8b49e9abb94\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" podUID="6b63cc5a-ba78-4672-b37a-d49c28673f50"
Jul 08 16:33:53.151884 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:53.151338    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:53.151884 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.151543    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:53.151884 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.151598    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:55.151582583 +0000 UTC m=+5.040997575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:53.252114 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:53.252029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:53.252258 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.252172    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:53.252258 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.252234    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:55.252217687 +0000 UTC m=+5.141632686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:53.352465 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:53.352424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:53.352609 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.352572    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:53.352609 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.352590    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:53.352609 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.352603    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:53.352794 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.352663    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:55.352645618 +0000 UTC m=+5.242060620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:53.578512 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:53.578404    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:28:51 +0000 UTC" deadline="2028-04-21 04:16:57.581914362 +0000 UTC"
Jul 08 16:33:53.578512 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:53.578438    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15659h43m4.003480949s"
Jul 08 16:33:53.669285 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:53.668716    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:53.669285 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:53.668841    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:33:54.081938 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:54.081907    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:54.673369 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:54.673340    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:54.673564 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:54.673536    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:33:54.673644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:54.673628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:54.673754 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:54.673727    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:33:55.169313 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:55.169202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:55.169313 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.169277    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:55.169897 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.169358    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:59.169337827 +0000 UTC m=+9.058752829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:55.270040 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:55.270009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:55.270200 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.270173    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:55.270258 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.270246    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:59.270225552 +0000 UTC m=+9.159640549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:55.370937 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:55.370903    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:55.371106 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.371036    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:55.371106 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.371053    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:55.371106 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.371061    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:55.371261 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.371112    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:59.371095187 +0000 UTC m=+9.260510190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:55.668733 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:55.668698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:55.668915 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:55.668818    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:33:56.668295 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:56.668265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:56.668782 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:56.668393    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:33:56.668782 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:56.668579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:56.668782 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:56.668718    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:33:57.668023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:57.667992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:57.668204 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:57.668127    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:33:58.673965 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:58.673928    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:58.674472 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:58.674062    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:33:58.674472 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:58.674087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:58.674472 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:58.674158    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:33:59.199237 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:59.199188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:59.199401 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.199344    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:59.199460 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.199411    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:07.199391999 +0000 UTC m=+17.088806990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:59.299969 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:59.299890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:33:59.300129 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.300056    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:59.300129 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.300112    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:07.30009282 +0000 UTC m=+17.189507823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:59.401085 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:59.401057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:33:59.401268 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.401253    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:59.401325 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.401296    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:59.401325 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.401309    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:59.401422 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.401365    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:07.401347253 +0000 UTC m=+17.290762244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:59.668695 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:33:59.668564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:33:59.668851 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:33:59.668712    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:00.671907 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:00.671879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:00.672276 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:00.671981    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:00.672276 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:00.672074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:00.672276 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:00.672178    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:01.668512 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.668306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:01.668642 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:01.668569    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:01.808930 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.808876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4x75t" event={"ID":"c29ba8ac-fe57-4260-9bee-92c4758bcbe5","Type":"ContainerStarted","Data":"5cfd61faf1468b0b58f0f0795a6c4543ae90de306e139cfb7f5350c1dd1f5c9c"}
Jul 08 16:34:01.810667 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.810637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4" event={"ID":"10de1d9b-34d5-45e6-8809-232627883766","Type":"ContainerStarted","Data":"e8402060eae7e1acebbe85cc48a1e89b92b84d5f5c0d3deb5d57555264f2ec0c"}
Jul 08 16:34:01.812197 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.812169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lhssb" event={"ID":"946032cc-95d9-4a74-ab85-10e9c5bc9051","Type":"ContainerStarted","Data":"ce5cff5f9e04ccf17644a4b00181b3f920f29e9e47ee0e36c7f795a563508286"}
Jul 08 16:34:01.813844 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.813814    2559 generic.go:356] "Generic (PLEG): container finished" podID="cc853c8f2f1ea6a968532ff9703fcfa8" containerID="14a8b91ae3c24dbebdd6cb43fcd39c566751f4041dfb8a85dfde39565a0ba394" exitCode=0
Jul 08 16:34:01.813926 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.813899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal" event={"ID":"cc853c8f2f1ea6a968532ff9703fcfa8","Type":"ContainerDied","Data":"14a8b91ae3c24dbebdd6cb43fcd39c566751f4041dfb8a85dfde39565a0ba394"}
Jul 08 16:34:01.815417 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.815398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdrjd" event={"ID":"90db286d-0405-4190-8aa9-42695ef44a41","Type":"ContainerStarted","Data":"ef8f284781b9a6728189ef5e091e3b61df2c2e8f35274e61978f935a328a9bf1"}
Jul 08 16:34:01.817003 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.816981    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbbf911d-083c-4b51-a7b4-b795abbc5e70" containerID="83b5cbc4993aea8752c6ef845c630ac206631c18f2a97359631298b36aefe931" exitCode=0
Jul 08 16:34:01.817087 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.817061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerDied","Data":"83b5cbc4993aea8752c6ef845c630ac206631c18f2a97359631298b36aefe931"}
Jul 08 16:34:01.818702 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.818650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sjfpx" event={"ID":"d19257c2-bde8-4433-bfdc-71e91f1c3caa","Type":"ContainerStarted","Data":"18a3f101d5d8e428148be01f8702f6eee2b56e02e56fc3a79bbc0c54e2a7fb00"}
Jul 08 16:34:01.820352 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.820330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal" event={"ID":"e23d70a312ac711b759fd35bb366a792","Type":"ContainerStarted","Data":"5b08c837606275b5d08ec388297c4f0611f4e1a1694495bbe16e36cac34d64f9"}
Jul 08 16:34:01.835695 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.835641    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4x75t" podStartSLOduration=1.7097728490000001 podStartE2EDuration="10.835627371s" totalImagesPullingTime="9.125854522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:01.835390794 +0000 UTC m=+11.724805813" watchObservedRunningTime="2026-07-08 16:34:01.835627371 +0000 UTC m=+11.725042382"
Jul 08 16:34:01.902172 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.902132    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lhssb" podStartSLOduration=2.714982256 podStartE2EDuration="11.90211262s" totalImagesPullingTime="9.187130364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:01.873209323 +0000 UTC m=+11.762624333" watchObservedRunningTime="2026-07-08 16:34:01.90211262 +0000 UTC m=+11.791527629"
Jul 08 16:34:01.951728 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.951635    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-118.ec2.internal" podStartSLOduration=10.951616247 podStartE2EDuration="10.951616247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:01.951482776 +0000 UTC m=+11.840897788" watchObservedRunningTime="2026-07-08 16:34:01.951616247 +0000 UTC m=+11.841031256"
Jul 08 16:34:01.951837 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.951720    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bdrjd" podStartSLOduration=1.8261615820000001 podStartE2EDuration="10.951713022s" totalImagesPullingTime="9.12555144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:01.926643787 +0000 UTC m=+11.816058788" watchObservedRunningTime="2026-07-08 16:34:01.951713022 +0000 UTC m=+11.841128035"
Jul 08 16:34:01.967308 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:01.967268    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sjfpx" podStartSLOduration=1.807675047 podStartE2EDuration="10.967254892s" totalImagesPullingTime="9.159579845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:01.967151394 +0000 UTC m=+11.856566406" watchObservedRunningTime="2026-07-08 16:34:01.967254892 +0000 UTC m=+11.856669902"
Jul 08 16:34:02.670673 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:02.670646    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:02.670872 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:02.670776    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:02.670872 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:02.670838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:02.673757 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:02.670943    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:02.823876 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:02.823788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-szs64" event={"ID":"61812833-1331-488b-96bd-e32d6fc6323e","Type":"ContainerStarted","Data":"7df73ff025c632c7de35fe503cde53e3b10db9e9b02bb7dc08aabfa79027d1a6"}
Jul 08 16:34:02.831827 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:02.831795    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:34:02.832842 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:02.832823    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:34:02.839505 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:02.839463    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-szs64" podStartSLOduration=3.630144213 podStartE2EDuration="12.839450901s" totalImagesPullingTime="9.209306688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:02.838951439 +0000 UTC m=+12.728366450" watchObservedRunningTime="2026-07-08 16:34:02.839450901 +0000 UTC m=+12.728865911"
Jul 08 16:34:03.668397 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:03.668154    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:03.668573 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:03.668504    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:04.670855 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:04.670828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:04.671321 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:04.670960    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:04.671321 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:04.671012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:04.671321 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:04.671120    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:04.827856 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:04.827829    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:34:05.668424 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:05.668393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:05.668545 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:05.668514    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:05.831738 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:05.831621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal" event={"ID":"cc853c8f2f1ea6a968532ff9703fcfa8","Type":"ContainerStarted","Data":"4bd9dfecaf02e119eaa8a234d0d0bf6c155e5c13db0ae84d188adcf2e6ace3c0"}
Jul 08 16:34:05.833299 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:05.833258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gz6lt" event={"ID":"dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa","Type":"ContainerStarted","Data":"a20ea2d17aa7ca5140e508fe356ea76591b41df6936b947de27d01b5e6127d20"}
Jul 08 16:34:05.849756 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:05.849711    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-118.ec2.internal" podStartSLOduration=14.84969369 podStartE2EDuration="14.84969369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:05.8490167 +0000 UTC m=+15.738431709" watchObservedRunningTime="2026-07-08 16:34:05.84969369 +0000 UTC m=+15.739108696"
Jul 08 16:34:05.866279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:05.866227    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gz6lt" podStartSLOduration=1.147134473 podStartE2EDuration="14.866206752s" totalImagesPullingTime="13.719072279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:05.865065235 +0000 UTC m=+15.754480244" watchObservedRunningTime="2026-07-08 16:34:05.866206752 +0000 UTC m=+15.755621784"
Jul 08 16:34:06.188977 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.188641    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:34:06.617003 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.616898    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:34:06.188899456Z","UUID":"3f887ce8-48e4-4e3d-8791-20a5006156e6","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:34:06.618214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.618180    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:34:06.618214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.618213    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:34:06.668656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.668632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:06.668795 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.668652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:06.668795 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:06.668760    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:06.668949 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:06.668926    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:06.837038 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:06.837002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4" event={"ID":"10de1d9b-34d5-45e6-8809-232627883766","Type":"ContainerStarted","Data":"7b0d4f062cd2f5c6ee5753514909e70737169fc208f6c5787d34b81ce6ec844b"}
Jul 08 16:34:07.256464 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:07.256428    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:07.256651 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.256590    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:07.256740 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.256671    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:23.256649925 +0000 UTC m=+33.146064915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:07.357508 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:07.357439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:07.357635 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.357578    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:07.357723 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.357645    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:23.357624519 +0000 UTC m=+33.247039520 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:07.458235 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:07.458204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:07.458406 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.458376    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:07.458406 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.458396    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:07.458406 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.458407    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:07.458558 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.458459    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:23.458441457 +0000 UTC m=+33.347856451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:07.668654 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:07.668584    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:07.668814 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:07.668698    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:07.843413 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:07.843378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4" event={"ID":"10de1d9b-34d5-45e6-8809-232627883766","Type":"ContainerStarted","Data":"dc81dc2c954ae073a871fef27743bd5f209c8263225eca30832c13d5dcd79086"}
Jul 08 16:34:07.862611 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:07.862563    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ksgm4" podStartSLOduration=2.237538347 podStartE2EDuration="16.862546679s" totalImagesPullingTime="14.625008332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:07.862299349 +0000 UTC m=+17.751714371" watchObservedRunningTime="2026-07-08 16:34:07.862546679 +0000 UTC m=+17.751961688"
Jul 08 16:34:08.668814 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:08.668775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:08.668985 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:08.668896    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:08.668985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:08.668783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:08.669115 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:08.669055    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:09.668286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:09.668253    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:09.668743 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:09.668388    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:10.668799 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:10.668775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:10.669146 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:10.668879    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:10.669146 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:10.668972    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:10.669146 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:10.669083    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:10.851158 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:10.851074    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbbf911d-083c-4b51-a7b4-b795abbc5e70" containerID="f4ed2b461996ee288c3584e2931e1e4a5961372b367b14ffb007e97173ad1689" exitCode=0
Jul 08 16:34:10.851158 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:10.851120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerDied","Data":"f4ed2b461996ee288c3584e2931e1e4a5961372b367b14ffb007e97173ad1689"}
Jul 08 16:34:11.668008 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:11.667981    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:11.668090 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:11.668074    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:11.855431 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:11.855357    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbbf911d-083c-4b51-a7b4-b795abbc5e70" containerID="96b7c2f59e6b2636a89221c1145f154a01ba2f1d184b8ffc5bd9a3350deeefa9" exitCode=0
Jul 08 16:34:11.855431 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:11.855402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerDied","Data":"96b7c2f59e6b2636a89221c1145f154a01ba2f1d184b8ffc5bd9a3350deeefa9"}
Jul 08 16:34:12.668473 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:12.668449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:12.668585 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:12.668449    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:12.668585 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:12.668568    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:12.668713 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:12.668664    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:12.859903 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:12.859819    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbbf911d-083c-4b51-a7b4-b795abbc5e70" containerID="03e77aabdf55d7d1fe97cfb26383ea63b52701bb0b28eccfeca59cf0a7593300" exitCode=0
Jul 08 16:34:12.859903 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:12.859883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerDied","Data":"03e77aabdf55d7d1fe97cfb26383ea63b52701bb0b28eccfeca59cf0a7593300"}
Jul 08 16:34:13.668800 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:13.668769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:13.668984 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:13.668894    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:14.637591 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:14.637560    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:34:14.638117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:14.637719    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:34:14.638218 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:14.638203    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdrjd"
Jul 08 16:34:14.668843 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:14.668811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:14.668991 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:14.668930    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:14.669052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:14.669007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:14.669150 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:14.669129    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:15.668214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:15.668182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:15.668605 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:15.668314    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:16.667880 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:16.667803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:16.667880 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:16.667847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:16.668104 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:16.667946    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:16.668104 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:16.668032    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:17.668408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:17.668379    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:17.668707 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:17.668467    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:17.872466 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:17.872441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerStarted","Data":"28f7bca22b7ab1d579feb46ab6cbbd598290d496bb4015a52a327ce1f27af3b5"}
Jul 08 16:34:18.668139 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:18.668105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:18.668346 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:18.668249    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:18.668346 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:18.668305    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:18.668462 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:18.668414    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:18.877694 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:18.877642    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbbf911d-083c-4b51-a7b4-b795abbc5e70" containerID="28f7bca22b7ab1d579feb46ab6cbbd598290d496bb4015a52a327ce1f27af3b5" exitCode=0
Jul 08 16:34:18.877877 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:18.877715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerDied","Data":"28f7bca22b7ab1d579feb46ab6cbbd598290d496bb4015a52a327ce1f27af3b5"}
Jul 08 16:34:19.668143 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:19.668112    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:19.668409 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:19.668213    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:20.669593 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:20.669562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:20.670068 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:20.669697    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:20.670068 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:20.669706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:20.670068 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:20.669828    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:21.668752 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:21.668718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:21.668937 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:21.668827    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:22.668715 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:22.668629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:22.669046 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:22.668788    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:22.669046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:22.668831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:22.669046 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:22.668930    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:22.888528 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:22.888505    2559 generic.go:356] "Generic (PLEG): container finished" podID="fbbf911d-083c-4b51-a7b4-b795abbc5e70" containerID="a935c42ca4cb6a129fcd94f9f563d1a72d11a98170f35c7a7ae1e3b610f72dc1" exitCode=0
Jul 08 16:34:22.888632 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:22.888571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerDied","Data":"a935c42ca4cb6a129fcd94f9f563d1a72d11a98170f35c7a7ae1e3b610f72dc1"}
Jul 08 16:34:22.890019 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:22.889999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"eef37ddddd160b5170be072fb87c9f988caebd4ca1c90d7364c912a798570bdc"}
Jul 08 16:34:23.274695 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.274509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:23.274807 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.274638    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:23.274807 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.274750    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret podName:598f3885-6ab6-4276-a3f0-8b6e634cf091 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:55.27473628 +0000 UTC m=+65.164151267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret") pod "global-pull-secret-syncer-kn2hm" (UID: "598f3885-6ab6-4276-a3f0-8b6e634cf091") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:23.375090 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.375068    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:23.375221 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.375158    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:23.375221 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.375197    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:55.375185821 +0000 UTC m=+65.264600813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:23.475673 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.475647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:23.475790 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.475776    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:23.475828 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.475792    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:23.475828 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.475801    2559 projected.go:196] Error preparing data for projected volume kube-api-access-s42qd for pod openshift-network-diagnostics/network-check-target-mlbdg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:23.475888 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.475841    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd podName:a8cffb6b-8aa7-482d-8ec2-e88558428be5 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:55.475829581 +0000 UTC m=+65.365244574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s42qd" (UniqueName: "kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd") pod "network-check-target-mlbdg" (UID: "a8cffb6b-8aa7-482d-8ec2-e88558428be5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:23.668104 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.668084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:23.668206 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:23.668168    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:23.894362 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894313    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:34:23.894964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894620    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b63cc5a-ba78-4672-b37a-d49c28673f50" containerID="e219d177fe456494257c1e5c2e177d47facf428c09ab964afd82a2f8fe425dd1" exitCode=1
Jul 08 16:34:23.894964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"c12bcff7fcbc12eae3fae8015ed60a042b6d3332d200f90330ef5e525015cfc7"}
Jul 08 16:34:23.894964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"f6484c9497113b0457cece1eb6bb362fdecafb92fb7fdf40be98c5b802bedd0c"}
Jul 08 16:34:23.894964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"4a4d9061270e03b1ea4955328eabba71c8d791bc25451bac9b9b5df8af4401fd"}
Jul 08 16:34:23.894964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"e1457251556c04dfb738978986532d1c53bf519e8a935bd9c78bc9cd429ad3c7"}
Jul 08 16:34:23.894964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.894861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerDied","Data":"e219d177fe456494257c1e5c2e177d47facf428c09ab964afd82a2f8fe425dd1"}
Jul 08 16:34:23.897888 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.897870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b45pc" event={"ID":"fbbf911d-083c-4b51-a7b4-b795abbc5e70","Type":"ContainerStarted","Data":"59da4a8287ed058d19c7dca456bf7a9964260f8a1e00f7eeabe8166cc9bc59ae"}
Jul 08 16:34:23.921580 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:23.921545    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b45pc" podStartSLOduration=8.935502052 podStartE2EDuration="32.921532771s" totalImagesPullingTime="23.986030719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:23.921504446 +0000 UTC m=+33.810919456" watchObservedRunningTime="2026-07-08 16:34:23.921532771 +0000 UTC m=+33.810947781"
Jul 08 16:34:24.668720 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:24.668689    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:24.668908 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:24.668799    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:24.668908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:24.668849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:24.669023 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:24.668940    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:25.668566 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:25.668494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:25.668938 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:25.668590    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:25.906147 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:25.906122    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:34:25.906586 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:25.906567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"2a824c9f640d6656dfcb543ecbb6203b5f68ac0d8fdb9985133c653337eae771"}
Jul 08 16:34:26.668133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:26.668102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:26.668308 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:26.668198    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:26.668308 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:26.668271    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:26.668426 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:26.668347    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:27.669049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:27.668798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:27.669418 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:27.669079    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:27.913982 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:27.913957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:34:27.914329 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:27.914310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"1309357438a3eda59d82c312c3764925c0deb93bfcb206e7b90ae808aa84f69c"}
Jul 08 16:34:27.914572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:27.914546    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:34:27.914735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:27.914721    2559 scope.go:117] "RemoveContainer" containerID="e219d177fe456494257c1e5c2e177d47facf428c09ab964afd82a2f8fe425dd1"
Jul 08 16:34:27.934316 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:27.934290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:34:28.668126 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.668079    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:28.668316 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.668097    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:28.668316 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:28.668224    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:28.668316 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:28.668252    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:28.920578 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.920516    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:34:28.920997 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.920897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" event={"ID":"6b63cc5a-ba78-4672-b37a-d49c28673f50","Type":"ContainerStarted","Data":"25e1f619875231da04baa5cfd7e0241ddc460a9d3eda0baeea7a767d8822ad70"}
Jul 08 16:34:28.921102 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.921080    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:34:28.921167 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.921154    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:34:28.935089 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.935068    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:34:28.951807 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:28.951766    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp" podStartSLOduration=7.192863618 podStartE2EDuration="37.951751763s" totalImagesPullingTime="30.758888145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-08 16:34:28.951406436 +0000 UTC m=+38.840821458" watchObservedRunningTime="2026-07-08 16:34:28.951751763 +0000 UTC m=+38.841166773"
Jul 08 16:34:29.589635 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:29.589605    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kn2hm"]
Jul 08 16:34:29.589791 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:29.589757    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:29.589899 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:29.589875    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:29.592618 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:29.592593    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-txdt2"]
Jul 08 16:34:29.592765 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:29.592688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:29.592819 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:29.592785    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:29.593300 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:29.593280    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mlbdg"]
Jul 08 16:34:29.593404 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:29.593364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:29.593476 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:29.593456    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:30.668512 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:30.668477    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:30.669014 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:30.668578    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:31.667991 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:31.667953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:31.668170 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:31.668070    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:31.668170 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:31.668120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:31.668246 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:31.668234    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:32.668754 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:32.668672    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:32.669098 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:32.668789    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:33.668276 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:33.668247    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:33.668435 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:33.668370    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:33.668435 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:33.668422    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:33.668560 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:33.668496    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:34.668115 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:34.668049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:34.668455 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:34.668140    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kn2hm" podUID="598f3885-6ab6-4276-a3f0-8b6e634cf091"
Jul 08 16:34:35.668467 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:35.668288    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:35.668863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:35.668324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:35.668863 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:35.668553    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-txdt2" podUID="33c52f87-ae67-40cb-8711-1a37de967be6"
Jul 08 16:34:35.668863 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:35.668598    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbdg" podUID="a8cffb6b-8aa7-482d-8ec2-e88558428be5"
Jul 08 16:34:36.444735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.444644    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-118.ec2.internal" event="NodeReady"
Jul 08 16:34:36.444869 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.444758    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:34:36.480331 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.480305    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68447587c9-pfqnx"]
Jul 08 16:34:36.482166 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.482149    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.484004 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.483978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 16:34:36.484104 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.484024    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-746tn\""
Jul 08 16:34:36.484201 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.484186    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 16:34:36.484261 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.484213    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 16:34:36.489063 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.489043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 16:34:36.492661 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.492636    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hlm6j"]
Jul 08 16:34:36.496619 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.496600    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68447587c9-pfqnx"]
Jul 08 16:34:36.496715 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.496706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.498528 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.498500    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:34:36.498640 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.498572    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rv4r\""
Jul 08 16:34:36.498640 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.498604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:34:36.501286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.501268    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mz246"]
Jul 08 16:34:36.503052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.503035    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.504380 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.504362    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlm6j"]
Jul 08 16:34:36.504955 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.504937    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:34:36.505167 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.505141    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:34:36.505307 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.505291    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s64wv\""
Jul 08 16:34:36.505445 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.505425    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:34:36.512332 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.512314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mz246"]
Jul 08 16:34:36.576867 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.576839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.576872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwz9\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-kube-api-access-wqwz9\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.576899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-registry-certificates\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.576922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-trusted-ca\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577169 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.576999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-installation-pull-secrets\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577169 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.577035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-image-registry-private-configuration\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577169 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.577067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c839748c-70cd-4055-972b-f9c232a0b040-ca-trust-extracted\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.577169 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.577131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-bound-sa-token\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.668323 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.668299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:36.670532 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.670514    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:34:36.677982 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.677963    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c839748c-70cd-4055-972b-f9c232a0b040-ca-trust-extracted\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678069 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-bound-sa-token\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678069 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ct9s\" (UniqueName: \"kubernetes.io/projected/64fe723a-e250-4875-a8db-4a2186f92a51-kube-api-access-5ct9s\") pod \"ingress-canary-mz246\" (UID: \"64fe723a-e250-4875-a8db-4a2186f92a51\") " pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.678069 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-config-volume\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.678189 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678189 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwz9\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-kube-api-access-wqwz9\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678189 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64fe723a-e250-4875-a8db-4a2186f92a51-cert\") pod \"ingress-canary-mz246\" (UID: \"64fe723a-e250-4875-a8db-4a2186f92a51\") " pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:36.678188    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:36.678201    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-registry-certificates\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-trusted-ca\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:36.678275    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:37.178260921 +0000 UTC m=+47.067675913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qptpz\" (UniqueName: \"kubernetes.io/projected/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-kube-api-access-qptpz\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678326    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c839748c-70cd-4055-972b-f9c232a0b040-ca-trust-extracted\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678335 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.678625 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-installation-pull-secrets\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678625 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-image-registry-private-configuration\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.678625 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-tmp-dir\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.678756 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.678742    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-registry-certificates\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.679063 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.679046    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-trusted-ca\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.682028 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.682010    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-installation-pull-secrets\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.682028 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.682026    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-image-registry-private-configuration\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.687992 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.687972    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwz9\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-kube-api-access-wqwz9\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.688087 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.688054    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-bound-sa-token\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:36.779202 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ct9s\" (UniqueName: \"kubernetes.io/projected/64fe723a-e250-4875-a8db-4a2186f92a51-kube-api-access-5ct9s\") pod \"ingress-canary-mz246\" (UID: \"64fe723a-e250-4875-a8db-4a2186f92a51\") " pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.779284 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-config-volume\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.779284 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64fe723a-e250-4875-a8db-4a2186f92a51-cert\") pod \"ingress-canary-mz246\" (UID: \"64fe723a-e250-4875-a8db-4a2186f92a51\") " pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.779284 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qptpz\" (UniqueName: \"kubernetes.io/projected/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-kube-api-access-qptpz\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.779397 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.779505 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-tmp-dir\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.779505 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:36.779475    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:36.779605 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:36.779528    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:34:37.279511374 +0000 UTC m=+47.168926365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:36.779744 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779724    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-config-volume\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.779800 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.779728    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-tmp-dir\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.781278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.781257    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64fe723a-e250-4875-a8db-4a2186f92a51-cert\") pod \"ingress-canary-mz246\" (UID: \"64fe723a-e250-4875-a8db-4a2186f92a51\") " pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.787652 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.787633    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qptpz\" (UniqueName: \"kubernetes.io/projected/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-kube-api-access-qptpz\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:36.787970 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.787953    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ct9s\" (UniqueName: \"kubernetes.io/projected/64fe723a-e250-4875-a8db-4a2186f92a51-kube-api-access-5ct9s\") pod \"ingress-canary-mz246\" (UID: \"64fe723a-e250-4875-a8db-4a2186f92a51\") " pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.812543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.812524    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mz246"
Jul 08 16:34:36.935810 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:36.935786    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mz246"]
Jul 08 16:34:36.938826 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:34:36.938803    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fe723a_e250_4875_a8db_4a2186f92a51.slice/crio-47a2ef1765894e19e4e52805f4adc5392dbebbc31274e3eac8324181f1a83ddc WatchSource:0}: Error finding container 47a2ef1765894e19e4e52805f4adc5392dbebbc31274e3eac8324181f1a83ddc: Status 404 returned error can't find the container with id 47a2ef1765894e19e4e52805f4adc5392dbebbc31274e3eac8324181f1a83ddc
Jul 08 16:34:37.182049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.181973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:37.182165 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:37.182076    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:37.182165 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:37.182093    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:34:37.182165 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:37.182153    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:38.182138921 +0000 UTC m=+48.071553909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:34:37.282439 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.282411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:37.282545 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:37.282535    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:37.282593 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:37.282581    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:34:38.282569347 +0000 UTC m=+48.171984338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:37.668145 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.668114    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:37.668333 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.668311    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:37.670275 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.670251    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:34:37.670275 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.670268    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9wkjp\""
Jul 08 16:34:37.670768 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.670735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:34:37.670768 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.670749    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:34:37.671190 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.670824    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwzcp\""
Jul 08 16:34:37.941855 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:37.941785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mz246" event={"ID":"64fe723a-e250-4875-a8db-4a2186f92a51","Type":"ContainerStarted","Data":"47a2ef1765894e19e4e52805f4adc5392dbebbc31274e3eac8324181f1a83ddc"}
Jul 08 16:34:38.188964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:38.188929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:38.189113 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:38.189092    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:38.189113 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:38.189111    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:34:38.189184 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:38.189165    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:40.189150107 +0000 UTC m=+50.078565094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:34:38.289391 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:38.289361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:38.289529 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:38.289501    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:38.289588 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:38.289561    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:34:40.289543191 +0000 UTC m=+50.178958192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:39.609499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:39.609462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:39.707257 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:39.707218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:39.913655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:39.913626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:40.204824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:40.204742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:40.204965 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:40.204870    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:40.204965 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:40.204882    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:34:40.204965 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:40.204930    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:44.204917299 +0000 UTC m=+54.094332290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:34:40.305206 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:40.305173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:40.305364 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:40.305310    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:40.305364 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:40.305363    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:34:44.305348731 +0000 UTC m=+54.194763723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:40.949027 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:40.948995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mz246" event={"ID":"64fe723a-e250-4875-a8db-4a2186f92a51","Type":"ContainerStarted","Data":"55159465e92b45fb5e176c5ba8cb461aca26e28bb5d9e8cf2fbcbde2f37b63ac"}
Jul 08 16:34:40.964190 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:40.964149    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mz246" podStartSLOduration=2.005815487 podStartE2EDuration="4.964136428s" totalImagesPullingTime="2.958320941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:40.964020349 +0000 UTC m=+50.853435359" watchObservedRunningTime="2026-07-08 16:34:40.964136428 +0000 UTC m=+50.853551437"
Jul 08 16:34:43.902004 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:43.901964    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:44.001021 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:44.000991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:44.230262 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:44.230236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:44.230374 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:44.230352    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:44.230374 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:44.230364    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:34:44.230439 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:44.230414    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.230400154 +0000 UTC m=+62.119815167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:34:44.331325 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:44.331299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:44.331468 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:44.331450    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:44.331531 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:44.331518    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.331500462 +0000 UTC m=+62.220915468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:44.980520 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:44.980484    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:45.071483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:45.071444    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:47.891931 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:47.891894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:52.151256 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:52.151222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:52.286339 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:52.286311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:34:52.286458 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:52.286419    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:52.286458 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:52.286430    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:34:52.286524 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:52.286474    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:08.286462358 +0000 UTC m=+78.175877351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:34:52.387419 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:52.387394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:34:52.387537 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:52.387526    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:52.387585 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:52.387575    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:35:08.387562926 +0000 UTC m=+78.276977917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:55.308880 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.308843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:55.310939 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.310917    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:34:55.321662 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.321634    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/598f3885-6ab6-4276-a3f0-8b6e634cf091-original-pull-secret\") pod \"global-pull-secret-syncer-kn2hm\" (UID: \"598f3885-6ab6-4276-a3f0-8b6e634cf091\") " pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:55.409651 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.409626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:34:55.411752 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.411735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:34:55.420124 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:55.420109    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 16:34:55.420195 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:34:55.420159    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs podName:33c52f87-ae67-40cb-8711-1a37de967be6 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:59.420145922 +0000 UTC m=+129.309560910 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs") pod "network-metrics-daemon-txdt2" (UID: "33c52f87-ae67-40cb-8711-1a37de967be6") : secret "metrics-daemon-secret" not found
Jul 08 16:34:55.510003 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.509981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:55.511907 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.511892    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:34:55.521905 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.521888    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:34:55.533204 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.533184    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42qd\" (UniqueName: \"kubernetes.io/projected/a8cffb6b-8aa7-482d-8ec2-e88558428be5-kube-api-access-s42qd\") pod \"network-check-target-mlbdg\" (UID: \"a8cffb6b-8aa7-482d-8ec2-e88558428be5\") " pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:55.577032 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.576986    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kn2hm"
Jul 08 16:34:55.691251 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.691229    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9wkjp\""
Jul 08 16:34:55.700405 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.700380    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:34:55.731357 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.731337    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kn2hm"]
Jul 08 16:34:55.745062 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:34:55.745041    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598f3885_6ab6_4276_a3f0_8b6e634cf091.slice/crio-3b086ad54bae3f33750931baaf7e57bea1b6aa65e9a3ebe45eca784aa957fc14 WatchSource:0}: Error finding container 3b086ad54bae3f33750931baaf7e57bea1b6aa65e9a3ebe45eca784aa957fc14: Status 404 returned error can't find the container with id 3b086ad54bae3f33750931baaf7e57bea1b6aa65e9a3ebe45eca784aa957fc14
Jul 08 16:34:55.837985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.837916    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mlbdg"]
Jul 08 16:34:55.842171 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:34:55.842150    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cffb6b_8aa7_482d_8ec2_e88558428be5.slice/crio-eebf30e73063af3112b3649e12408b8765133764db36ac06bd7f37934d7e146a WatchSource:0}: Error finding container eebf30e73063af3112b3649e12408b8765133764db36ac06bd7f37934d7e146a: Status 404 returned error can't find the container with id eebf30e73063af3112b3649e12408b8765133764db36ac06bd7f37934d7e146a
Jul 08 16:34:55.981247 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.981214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mlbdg" event={"ID":"a8cffb6b-8aa7-482d-8ec2-e88558428be5","Type":"ContainerStarted","Data":"eebf30e73063af3112b3649e12408b8765133764db36ac06bd7f37934d7e146a"}
Jul 08 16:34:55.982178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:55.982153    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kn2hm" event={"ID":"598f3885-6ab6-4276-a3f0-8b6e634cf091","Type":"ContainerStarted","Data":"3b086ad54bae3f33750931baaf7e57bea1b6aa65e9a3ebe45eca784aa957fc14"}
Jul 08 16:34:56.377066 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:56.377011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:56.478280 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:56.478251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:56.693708 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:56.693618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:56.794329 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:56.794257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:56.903073 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:56.903029    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:59.994032 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:59.993999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mlbdg" event={"ID":"a8cffb6b-8aa7-482d-8ec2-e88558428be5","Type":"ContainerStarted","Data":"afaf942b0926b6a8109a073d4f6ae81985c82115de998006487e70aa1592130a"}
Jul 08 16:34:59.994482 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:34:59.994148    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:35:00.010634 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:00.010587    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mlbdg" podStartSLOduration=65.27525126 podStartE2EDuration="1m9.010570574s" totalImagesPullingTime="3.735319314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:00.009701031 +0000 UTC m=+69.899116043" watchObservedRunningTime="2026-07-08 16:35:00.010570574 +0000 UTC m=+69.899985584"
Jul 08 16:35:00.910095 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:00.910054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:00.936112 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:00.936090    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xbnrp"
Jul 08 16:35:01.220085 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:01.220005    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.306023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:01.305750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.405124 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:01.405088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.999958 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:01.999922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kn2hm" event={"ID":"598f3885-6ab6-4276-a3f0-8b6e634cf091","Type":"ContainerStarted","Data":"21fc89d41771c1c3ba1c49bde71b7e3f08cf9901023b976faed878f9f7db2692"}
Jul 08 16:35:02.016139 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:02.016091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kn2hm" podStartSLOduration=65.527144302 podStartE2EDuration="1m11.016073728s" totalImagesPullingTime="5.488929426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:02.015954417 +0000 UTC m=+71.905369428" watchObservedRunningTime="2026-07-08 16:35:02.016073728 +0000 UTC m=+71.905488733"
Jul 08 16:35:04.061279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:04.061247    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.162691 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:04.162652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.648762 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.648734    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"]
Jul 08 16:35:07.652992 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.652976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:07.655982 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.655964    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 08 16:35:07.656472 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.656454    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-26xgz\""
Jul 08 16:35:07.656830 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.656816    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:35:07.657387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.657373    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 08 16:35:07.678781 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.678759    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"]
Jul 08 16:35:07.695474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.695454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwb5\" (UniqueName: \"kubernetes.io/projected/e7de2636-2a65-4342-ad36-2c002de6353d-kube-api-access-bcwb5\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:07.695586 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.695547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:07.796341 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.796323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:07.796448 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.796358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwb5\" (UniqueName: \"kubernetes.io/projected/e7de2636-2a65-4342-ad36-2c002de6353d-kube-api-access-bcwb5\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:07.796493 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:07.796462    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:07.796537 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:07.796523    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls podName:e7de2636-2a65-4342-ad36-2c002de6353d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:08.296508849 +0000 UTC m=+78.185923838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nb68s" (UID: "e7de2636-2a65-4342-ad36-2c002de6353d") : secret "samples-operator-tls" not found
Jul 08 16:35:07.802966 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.802944    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"]
Jul 08 16:35:07.805688 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.805665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"
Jul 08 16:35:07.812141 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.812121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ckdjd\""
Jul 08 16:35:07.812379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.812365    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:35:07.812596 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.812582    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 08 16:35:07.826884 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.826867    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwb5\" (UniqueName: \"kubernetes.io/projected/e7de2636-2a65-4342-ad36-2c002de6353d-kube-api-access-bcwb5\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:07.832570 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.832545    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"]
Jul 08 16:35:07.897124 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.897099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6h8\" (UniqueName: \"kubernetes.io/projected/da4a3ca2-873f-4659-80f1-829ae85b4edf-kube-api-access-xt6h8\") pod \"volume-data-source-validator-5d456fdbcd-5wjj5\" (UID: \"da4a3ca2-873f-4659-80f1-829ae85b4edf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"
Jul 08 16:35:07.998411 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:07.998388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6h8\" (UniqueName: \"kubernetes.io/projected/da4a3ca2-873f-4659-80f1-829ae85b4edf-kube-api-access-xt6h8\") pod \"volume-data-source-validator-5d456fdbcd-5wjj5\" (UID: \"da4a3ca2-873f-4659-80f1-829ae85b4edf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"
Jul 08 16:35:08.011863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.011836    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6h8\" (UniqueName: \"kubernetes.io/projected/da4a3ca2-873f-4659-80f1-829ae85b4edf-kube-api-access-xt6h8\") pod \"volume-data-source-validator-5d456fdbcd-5wjj5\" (UID: \"da4a3ca2-873f-4659-80f1-829ae85b4edf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"
Jul 08 16:35:08.086607 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.086579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:08.113636 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.113616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"
Jul 08 16:35:08.232149 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.232044    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5"]
Jul 08 16:35:08.234580 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:08.234556    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4a3ca2_873f_4659_80f1_829ae85b4edf.slice/crio-2399f762012d483d1d8a8961e7d98b1cc39393023509368312dce39ec1db26e9 WatchSource:0}: Error finding container 2399f762012d483d1d8a8961e7d98b1cc39393023509368312dce39ec1db26e9: Status 404 returned error can't find the container with id 2399f762012d483d1d8a8961e7d98b1cc39393023509368312dce39ec1db26e9
Jul 08 16:35:08.300903 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.300843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:08.300903 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.300877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:35:08.301022 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.300981    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:08.301022 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.300990    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68447587c9-pfqnx: secret "image-registry-tls" not found
Jul 08 16:35:08.301022 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.301000    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:08.301128 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.301033    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls podName:c839748c-70cd-4055-972b-f9c232a0b040 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:40.301019493 +0000 UTC m=+110.190434486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls") pod "image-registry-68447587c9-pfqnx" (UID: "c839748c-70cd-4055-972b-f9c232a0b040") : secret "image-registry-tls" not found
Jul 08 16:35:08.301128 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.301054    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls podName:e7de2636-2a65-4342-ad36-2c002de6353d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:09.301041055 +0000 UTC m=+79.190456046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nb68s" (UID: "e7de2636-2a65-4342-ad36-2c002de6353d") : secret "samples-operator-tls" not found
Jul 08 16:35:08.401310 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.401286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:35:08.401432 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.401415    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:08.401477 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:08.401463    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls podName:b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca nodeName:}" failed. No retries permitted until 2026-07-08 16:35:40.401451883 +0000 UTC m=+110.290866870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls") pod "dns-default-hlm6j" (UID: "b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:08.480644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:08.480625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:09.018040 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:09.018008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5" event={"ID":"da4a3ca2-873f-4659-80f1-829ae85b4edf","Type":"ContainerStarted","Data":"2399f762012d483d1d8a8961e7d98b1cc39393023509368312dce39ec1db26e9"}
Jul 08 16:35:09.308622 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:09.308562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:09.308744 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:09.308698    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:09.308791 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:09.308764    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls podName:e7de2636-2a65-4342-ad36-2c002de6353d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:11.308749444 +0000 UTC m=+81.198164435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nb68s" (UID: "e7de2636-2a65-4342-ad36-2c002de6353d") : secret "samples-operator-tls" not found
Jul 08 16:35:10.021959 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:10.021926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5" event={"ID":"da4a3ca2-873f-4659-80f1-829ae85b4edf","Type":"ContainerStarted","Data":"9de4e37db8cf05c96b69f4e7f1515f72de2e11f8dabe73c6ca95765ec8919af6"}
Jul 08 16:35:10.042361 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:10.042316    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-5wjj5" podStartSLOduration=2.084447288 podStartE2EDuration="3.042301445s" totalImagesPullingTime="957.854157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:10.041433495 +0000 UTC m=+79.930848505" watchObservedRunningTime="2026-07-08 16:35:10.042301445 +0000 UTC m=+79.931716454"
Jul 08 16:35:10.146416 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:10.146387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.218871 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:10.218847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.322266 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:11.322233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:11.322737 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:11.322397    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:11.322737 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:11.322483    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls podName:e7de2636-2a65-4342-ad36-2c002de6353d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:15.32246322 +0000 UTC m=+85.211878221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nb68s" (UID: "e7de2636-2a65-4342-ad36-2c002de6353d") : secret "samples-operator-tls" not found
Jul 08 16:35:14.793337 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:14.793209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.891420 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:14.891395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.980985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:14.980962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.150562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.150472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.320926 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.320885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.348006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.347984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:15.348139 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:15.348125    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:15.348187 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:15.348178    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls podName:e7de2636-2a65-4342-ad36-2c002de6353d nodeName:}" failed. No retries permitted until 2026-07-08 16:35:23.348165852 +0000 UTC m=+93.237580843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nb68s" (UID: "e7de2636-2a65-4342-ad36-2c002de6353d") : secret "samples-operator-tls" not found
Jul 08 16:35:15.365556 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.365530    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h242d"]
Jul 08 16:35:15.372433 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.372411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.374240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.374221    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:35:15.374824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.374807    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:35:15.374917 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.374829    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-82p44\""
Jul 08 16:35:15.374917 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.374868    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:35:15.374917 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.374883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:35:15.379177 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.378868    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h242d"]
Jul 08 16:35:15.417370 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.417318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.448392 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.448371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73409722-b9d0-4343-8331-9bc7e60d5f2e-crio-socket\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.448460 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.448400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.448460 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.448422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.448460 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.448447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm22c\" (UniqueName: \"kubernetes.io/projected/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-api-access-cm22c\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.448558 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.448529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73409722-b9d0-4343-8331-9bc7e60d5f2e-data-volume\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.481633 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.481610    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.549038 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73409722-b9d0-4343-8331-9bc7e60d5f2e-crio-socket\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.549140 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.549140 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.549140 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm22c\" (UniqueName: \"kubernetes.io/projected/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-api-access-cm22c\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.549140 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73409722-b9d0-4343-8331-9bc7e60d5f2e-data-volume\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.549140 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:15.549135    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:15.549368 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:15.549200    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:15.549368 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:15.549209    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:16.049192127 +0000 UTC m=+85.938607135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:15.549368 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549134    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73409722-b9d0-4343-8331-9bc7e60d5f2e-crio-socket\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.549368 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:15.549264    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:16.049247001 +0000 UTC m=+85.938661992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:15.549496 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.549379    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73409722-b9d0-4343-8331-9bc7e60d5f2e-data-volume\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.557506 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.557490    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm22c\" (UniqueName: \"kubernetes.io/projected/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-api-access-cm22c\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:15.669790 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.669724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.769497 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.769457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.871748 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.871721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.973079 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:15.973020    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.052195 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:16.052176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:16.052301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:16.052210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:16.052368 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:16.052302    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:16.052368 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:16.052360    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:16.052461 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:16.052380    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:17.052359926 +0000 UTC m=+86.941774926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:16.052461 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:16.052408    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:17.052396217 +0000 UTC m=+86.941811224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:17.058541 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.058508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:17.058541 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.058545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:17.058964 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:17.058615    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:17.058964 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:17.058643    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:17.058964 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:17.058675    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:19.05865974 +0000 UTC m=+88.948074747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:17.058964 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:17.058707    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:19.058698676 +0000 UTC m=+88.948113676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:17.114950 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.114915    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.213494 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.213463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.458185 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.458120    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-wvnqg"]
Jul 08 16:35:17.461069 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.461055    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.462940 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.462913    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 08 16:35:17.463050 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.462957    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 08 16:35:17.463529 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.463504    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 08 16:35:17.463529 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.463525    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 08 16:35:17.463651 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.463549    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jl45m\""
Jul 08 16:35:17.469382 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.469360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-wvnqg"]
Jul 08 16:35:17.493491 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.493471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.561351 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.561330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0941f7da-4753-4310-b988-cacf316a2434-signing-key\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.561408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.561369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0941f7da-4753-4310-b988-cacf316a2434-signing-cabundle\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.561495 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.561476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mb48\" (UniqueName: \"kubernetes.io/projected/0941f7da-4753-4310-b988-cacf316a2434-kube-api-access-9mb48\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.570548 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.570526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.662743 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.662723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mb48\" (UniqueName: \"kubernetes.io/projected/0941f7da-4753-4310-b988-cacf316a2434-kube-api-access-9mb48\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.662824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.662762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0941f7da-4753-4310-b988-cacf316a2434-signing-key\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.662824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.662798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0941f7da-4753-4310-b988-cacf316a2434-signing-cabundle\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.664046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.664030    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0941f7da-4753-4310-b988-cacf316a2434-signing-cabundle\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.665106 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.665085    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0941f7da-4753-4310-b988-cacf316a2434-signing-key\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.671329 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.671309    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mb48\" (UniqueName: \"kubernetes.io/projected/0941f7da-4753-4310-b988-cacf316a2434-kube-api-access-9mb48\") pod \"service-ca-758f599c5-wvnqg\" (UID: \"0941f7da-4753-4310-b988-cacf316a2434\") " pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.770906 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.770884    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-wvnqg"
Jul 08 16:35:17.890962 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:17.890934    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-wvnqg"]
Jul 08 16:35:17.893743 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:17.893717    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0941f7da_4753_4310_b988_cacf316a2434.slice/crio-8f8ca3dcb352828cb926de3e2927df5663919613259102205f5796b3253fb5fb WatchSource:0}: Error finding container 8f8ca3dcb352828cb926de3e2927df5663919613259102205f5796b3253fb5fb: Status 404 returned error can't find the container with id 8f8ca3dcb352828cb926de3e2927df5663919613259102205f5796b3253fb5fb
Jul 08 16:35:18.040357 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:18.040293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-wvnqg" event={"ID":"0941f7da-4753-4310-b988-cacf316a2434","Type":"ContainerStarted","Data":"8f8ca3dcb352828cb926de3e2927df5663919613259102205f5796b3253fb5fb"}
Jul 08 16:35:18.365035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:18.364968    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhssb_946032cc-95d9-4a74-ab85-10e9c5bc9051/dns-node-resolver/0.log"
Jul 08 16:35:18.613355 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:18.613319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:19.072867 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:19.072838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:19.073044 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:19.072882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:19.073044 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:19.072972    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:19.073044 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:19.073042    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:23.073022513 +0000 UTC m=+92.962437524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:19.073178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:19.073043    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:19.073178 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:19.073098    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:23.073085911 +0000 UTC m=+92.962500904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:19.363829 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:19.363754    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sjfpx_d19257c2-bde8-4433-bfdc-71e91f1c3caa/node-ca/0.log"
Jul 08 16:35:20.048214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:20.048176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-wvnqg" event={"ID":"0941f7da-4753-4310-b988-cacf316a2434","Type":"ContainerStarted","Data":"7472c8a7bfa56346401e1837443943ce6ccca5e9a0c1bac069e029791a17b53c"}
Jul 08 16:35:20.167071 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:20.167041    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mz246_64fe723a-e250-4875-a8db-4a2186f92a51/serve-healthcheck-canary/0.log"
Jul 08 16:35:21.101432 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:21.101391    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:21.201951 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:21.201925    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:21.301867 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:21.301821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:21.406851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:21.406768    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:21.502903 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:21.502873    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:21.603674 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:21.603641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.000285 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:22.000252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.405238 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:22.405159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:23.005278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.005248    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:23.104501 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.104468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:23.104651 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.104523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:23.104651 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:23.104591    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:23.104651 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:23.104624    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:23.104831 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:23.104659    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:31.104644556 +0000 UTC m=+100.994059553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:23.104831 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:23.104708    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:31.10466913 +0000 UTC m=+100.994084122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:23.407283 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.407213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:23.409469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.409437    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7de2636-2a65-4342-ad36-2c002de6353d-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nb68s\" (UID: \"e7de2636-2a65-4342-ad36-2c002de6353d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:23.561157 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.561120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"
Jul 08 16:35:23.603532 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.603494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:23.682773 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.682555    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-wvnqg" podStartSLOduration=5.039027422 podStartE2EDuration="6.682538708s" totalImagesPullingTime="1.643511286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:20.074174175 +0000 UTC m=+89.963589184" watchObservedRunningTime="2026-07-08 16:35:23.682538708 +0000 UTC m=+93.571953718"
Jul 08 16:35:23.683082 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.683066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s"]
Jul 08 16:35:23.795983 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:23.795958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.062404 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:24.062375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s" event={"ID":"e7de2636-2a65-4342-ad36-2c002de6353d","Type":"ContainerStarted","Data":"58f7b407b6b5e3fa83e91a199ad2cbd8ef1f7e346b16ccda5554b95f20155376"}
Jul 08 16:35:24.196963 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:24.196899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.396964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:24.396825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.713543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:24.713451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.815096 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:24.815026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.915052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:24.915024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:25.205839 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:25.205802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:25.306648 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:25.306613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:25.797465 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:25.797438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:26.069452 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:26.069380    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s" event={"ID":"e7de2636-2a65-4342-ad36-2c002de6353d","Type":"ContainerStarted","Data":"6f666090a0855f60d4ebf31fed3b18c68a5f1798bfa24542a5af3074d144866a"}
Jul 08 16:35:26.069452 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:26.069445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s" event={"ID":"e7de2636-2a65-4342-ad36-2c002de6353d","Type":"ContainerStarted","Data":"b10e732d7a77d2d3f6c1f5c5cda70e9c2f5ad57e241e2459dec3c38f374f10d3"}
Jul 08 16:35:26.086734 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:26.086671    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nb68s" podStartSLOduration=17.417376697999998 podStartE2EDuration="19.086656953s" totalImagesPullingTime="1.669280255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:26.085470011 +0000 UTC m=+95.974885021" watchObservedRunningTime="2026-07-08 16:35:26.086656953 +0000 UTC m=+95.976071962"
Jul 08 16:35:26.578813 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:26.578773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:26.679978 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:26.679949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.009263 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:27.009233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.222362 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:27.222330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.743798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:27.743757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.840069 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:27.840042    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:27.988010 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:27.987970    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.247871 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:28.247842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.404054 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:28.404022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:28.999881 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:28.999850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:29.407710 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:29.407619    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:29.805577 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:29.805543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:29.996614 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:29.996568    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:30.659448 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:30.659410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:30.732736 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:30.732699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:30.999460 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:30.999437    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mlbdg"
Jul 08 16:35:31.003323 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.003298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.169725 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.169697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:31.169872 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.169738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:31.169943 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:31.169887    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:31.169999 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:35:31.169952    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls podName:73409722-b9d0-4343-8331-9bc7e60d5f2e nodeName:}" failed. No retries permitted until 2026-07-08 16:35:47.16993643 +0000 UTC m=+117.059351428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h242d" (UID: "73409722-b9d0-4343-8331-9bc7e60d5f2e") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:31.170255 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.170234    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73409722-b9d0-4343-8331-9bc7e60d5f2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:31.200338 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.200312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.702346 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.702315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.831692 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:31.831658    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.109914 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:32.109882    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.365672 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:32.365582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.462006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:32.461979    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.775567 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:32.775534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.849598 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:32.849574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.974214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:32.974186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.038886 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:33.038814    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.192653 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:33.192608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.784801 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:33.784762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.892795 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:33.892763    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.995183 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:33.995144    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.715269 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:34.715241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.781534 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:34.781500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.289822 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.289789    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"]
Jul 08 16:35:36.325000 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.324970    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"]
Jul 08 16:35:36.325134 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.325112    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.327437 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.327414    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 16:35:36.327560 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.327458    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 16:35:36.328111 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.328081    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 08 16:35:36.328237 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.328170    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 16:35:36.328502 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.328488    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lwlct\""
Jul 08 16:35:36.338769 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.338750    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"]
Jul 08 16:35:36.338864 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.338776    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"]
Jul 08 16:35:36.338864 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.338846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.340511 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.340498    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 08 16:35:36.405569 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.405545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b8265cf-5fc8-4ead-a208-c273d1973668-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz\" (UID: \"3b8265cf-5fc8-4ead-a208-c273d1973668\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.405670 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.405581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq746\" (UniqueName: \"kubernetes.io/projected/f81814f1-5b62-4485-aca4-0a0370e3efc0-kube-api-access-zq746\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.405735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.405667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t6bk\" (UniqueName: \"kubernetes.io/projected/3b8265cf-5fc8-4ead-a208-c273d1973668-kube-api-access-5t6bk\") pod \"managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz\" (UID: \"3b8265cf-5fc8-4ead-a208-c273d1973668\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.405735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.405715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f81814f1-5b62-4485-aca4-0a0370e3efc0-klusterlet-config\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.405801 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.405734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f81814f1-5b62-4485-aca4-0a0370e3efc0-tmp\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.488787 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.488764    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.506940 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.506918    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b8265cf-5fc8-4ead-a208-c273d1973668-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz\" (UID: \"3b8265cf-5fc8-4ead-a208-c273d1973668\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.507038 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.506950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq746\" (UniqueName: \"kubernetes.io/projected/f81814f1-5b62-4485-aca4-0a0370e3efc0-kube-api-access-zq746\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.507108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.507087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t6bk\" (UniqueName: \"kubernetes.io/projected/3b8265cf-5fc8-4ead-a208-c273d1973668-kube-api-access-5t6bk\") pod \"managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz\" (UID: \"3b8265cf-5fc8-4ead-a208-c273d1973668\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.507154 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.507144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f81814f1-5b62-4485-aca4-0a0370e3efc0-klusterlet-config\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.507198 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.507177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f81814f1-5b62-4485-aca4-0a0370e3efc0-tmp\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.507542 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.507524    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f81814f1-5b62-4485-aca4-0a0370e3efc0-tmp\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.509429 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.509413    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f81814f1-5b62-4485-aca4-0a0370e3efc0-klusterlet-config\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.509512 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.509488    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3b8265cf-5fc8-4ead-a208-c273d1973668-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz\" (UID: \"3b8265cf-5fc8-4ead-a208-c273d1973668\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.518773 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.518752    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq746\" (UniqueName: \"kubernetes.io/projected/f81814f1-5b62-4485-aca4-0a0370e3efc0-kube-api-access-zq746\") pod \"klusterlet-addon-workmgr-85d9fb78bb-7b9pt\" (UID: \"f81814f1-5b62-4485-aca4-0a0370e3efc0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.518858 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.518775    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t6bk\" (UniqueName: \"kubernetes.io/projected/3b8265cf-5fc8-4ead-a208-c273d1973668-kube-api-access-5t6bk\") pod \"managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz\" (UID: \"3b8265cf-5fc8-4ead-a208-c273d1973668\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.586036 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.585982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.646436 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.646413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"
Jul 08 16:35:36.652850 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.652830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:36.682988 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.682948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.783893 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.783856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.801120 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.801038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz"]
Jul 08 16:35:36.803587 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:36.803564    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b8265cf_5fc8_4ead_a208_c273d1973668.slice/crio-6863c3789b34c7115c3ab76b34a2a5ee69ca92739dd113bca189fcd33f240d9e WatchSource:0}: Error finding container 6863c3789b34c7115c3ab76b34a2a5ee69ca92739dd113bca189fcd33f240d9e: Status 404 returned error can't find the container with id 6863c3789b34c7115c3ab76b34a2a5ee69ca92739dd113bca189fcd33f240d9e
Jul 08 16:35:36.834150 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:36.834128    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"]
Jul 08 16:35:36.836511 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:36.836469    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81814f1_5b62_4485_aca4_0a0370e3efc0.slice/crio-fea14c3f5853a776f6ddbf03f520ca41c38553f688d9543919b73bcd9754f3ab WatchSource:0}: Error finding container fea14c3f5853a776f6ddbf03f520ca41c38553f688d9543919b73bcd9754f3ab: Status 404 returned error can't find the container with id fea14c3f5853a776f6ddbf03f520ca41c38553f688d9543919b73bcd9754f3ab
Jul 08 16:35:37.101781 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:37.101698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt" event={"ID":"f81814f1-5b62-4485-aca4-0a0370e3efc0","Type":"ContainerStarted","Data":"fea14c3f5853a776f6ddbf03f520ca41c38553f688d9543919b73bcd9754f3ab"}
Jul 08 16:35:37.102700 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:37.102661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz" event={"ID":"3b8265cf-5fc8-4ead-a208-c273d1973668","Type":"ContainerStarted","Data":"6863c3789b34c7115c3ab76b34a2a5ee69ca92739dd113bca189fcd33f240d9e"}
Jul 08 16:35:39.774899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:39.774827    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.840100 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:39.840016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:40.117619 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.117534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz" event={"ID":"3b8265cf-5fc8-4ead-a208-c273d1973668","Type":"ContainerStarted","Data":"f852034f96d45c9fe9784eeaef63dd10da0466e46d3bb02618faa3bfa931b788"}
Jul 08 16:35:40.121227 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.121196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:40.133504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.133464    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d6c85b8b-8fsfz" podStartSLOduration=1.372653149 podStartE2EDuration="4.13345051s" totalImagesPullingTime="2.760797361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:40.132460191 +0000 UTC m=+110.021875203" watchObservedRunningTime="2026-07-08 16:35:40.13345051 +0000 UTC m=+110.022865519"
Jul 08 16:35:40.221116 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.221081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:40.340568 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.340534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:35:40.343210 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.343185    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"image-registry-68447587c9-pfqnx\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") " pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:35:40.394447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.394379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-746tn\""
Jul 08 16:35:40.403160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.403135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:35:40.441954 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.441928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:35:40.444419 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.444395    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca-metrics-tls\") pod \"dns-default-hlm6j\" (UID: \"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca\") " pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:35:40.708266 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.708196    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5rv4r\""
Jul 08 16:35:40.716775 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.716743    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:35:40.946698 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.946659    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68447587c9-pfqnx"]
Jul 08 16:35:40.949463 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:40.949441    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc839748c_70cd_4055_972b_f9c232a0b040.slice/crio-5514d43ed68aa5a3db61123f5619fcbce8e98ebdc4aa1d4c977df5e904cfc856 WatchSource:0}: Error finding container 5514d43ed68aa5a3db61123f5619fcbce8e98ebdc4aa1d4c977df5e904cfc856: Status 404 returned error can't find the container with id 5514d43ed68aa5a3db61123f5619fcbce8e98ebdc4aa1d4c977df5e904cfc856
Jul 08 16:35:40.965377 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:40.965325    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlm6j"]
Jul 08 16:35:40.967878 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:40.967794    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e8ff94_3fcc_4a12_adb9_90ebd08e67ca.slice/crio-d7ca0355795ddb54e984c248e87dd8d8584f1b942a618e437dd8f3e5c3ae3aab WatchSource:0}: Error finding container d7ca0355795ddb54e984c248e87dd8d8584f1b942a618e437dd8f3e5c3ae3aab: Status 404 returned error can't find the container with id d7ca0355795ddb54e984c248e87dd8d8584f1b942a618e437dd8f3e5c3ae3aab
Jul 08 16:35:41.122378 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.122339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68447587c9-pfqnx" event={"ID":"c839748c-70cd-4055-972b-f9c232a0b040","Type":"ContainerStarted","Data":"39f0397ce369b4e02081b6b787111a379fd1f50defc80a1eb5b3cef026b6bfb3"}
Jul 08 16:35:41.122545 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.122400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68447587c9-pfqnx" event={"ID":"c839748c-70cd-4055-972b-f9c232a0b040","Type":"ContainerStarted","Data":"5514d43ed68aa5a3db61123f5619fcbce8e98ebdc4aa1d4c977df5e904cfc856"}
Jul 08 16:35:41.122545 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.122427    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:35:41.123661 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.123631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlm6j" event={"ID":"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca","Type":"ContainerStarted","Data":"d7ca0355795ddb54e984c248e87dd8d8584f1b942a618e437dd8f3e5c3ae3aab"}
Jul 08 16:35:41.125041 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.125017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt" event={"ID":"f81814f1-5b62-4485-aca4-0a0370e3efc0","Type":"ContainerStarted","Data":"2bd8a70d0fa365e26e352708be86339588fe49962d8b6587287c910cadce981e"}
Jul 08 16:35:41.125228 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.125213    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:41.126798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.126780    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt"
Jul 08 16:35:41.141831 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.141788    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68447587c9-pfqnx" podStartSLOduration=110.141773965 podStartE2EDuration="1m50.141773965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:41.140527094 +0000 UTC m=+111.029942105" watchObservedRunningTime="2026-07-08 16:35:41.141773965 +0000 UTC m=+111.031188974"
Jul 08 16:35:41.155480 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.155442    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-85d9fb78bb-7b9pt" podStartSLOduration=1.105473462 podStartE2EDuration="5.155428997s" totalImagesPullingTime="4.049955535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:41.155371838 +0000 UTC m=+111.044786847" watchObservedRunningTime="2026-07-08 16:35:41.155428997 +0000 UTC m=+111.044843985"
Jul 08 16:35:41.906149 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.906096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:41.984058 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:41.984026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:42.086960 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:42.086911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:42.405301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:42.405264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.136727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:43.136676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlm6j" event={"ID":"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca","Type":"ContainerStarted","Data":"d863db3027d35466187ee3ef1c2f23a0ccb77a463e52b4e244f52a72fe6f057c"}
Jul 08 16:35:43.137101 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:43.136769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlm6j" event={"ID":"b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca","Type":"ContainerStarted","Data":"50c47b373659af49951e9d39cc34ea9680276f1e00e213c6de0356e541c6d3b9"}
Jul 08 16:35:43.156598 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:43.156550    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hlm6j" podStartSLOduration=65.67770493 podStartE2EDuration="1m7.156532458s" totalImagesPullingTime="1.478827528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:43.155768341 +0000 UTC m=+113.045183350" watchObservedRunningTime="2026-07-08 16:35:43.156532458 +0000 UTC m=+113.045947472"
Jul 08 16:35:44.140048 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:44.140017    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:35:45.121616 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:45.121577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:45.646858 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:45.646808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:45.746941 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:45.746889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.189171 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:47.189137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:47.191354 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:47.191334    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73409722-b9d0-4343-8331-9bc7e60d5f2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h242d\" (UID: \"73409722-b9d0-4343-8331-9bc7e60d5f2e\") " pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:47.482201 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:47.482180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h242d"
Jul 08 16:35:47.604863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:47.604833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h242d"]
Jul 08 16:35:47.607620 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:47.607595    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73409722_b9d0_4343_8331_9bc7e60d5f2e.slice/crio-10bb92ade7ef863731c71e231d99ff2176701f34f1df33149c14cc3d683beef1 WatchSource:0}: Error finding container 10bb92ade7ef863731c71e231d99ff2176701f34f1df33149c14cc3d683beef1: Status 404 returned error can't find the container with id 10bb92ade7ef863731c71e231d99ff2176701f34f1df33149c14cc3d683beef1
Jul 08 16:35:48.155754 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.155713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h242d" event={"ID":"73409722-b9d0-4343-8331-9bc7e60d5f2e","Type":"ContainerStarted","Data":"10bb92ade7ef863731c71e231d99ff2176701f34f1df33149c14cc3d683beef1"}
Jul 08 16:35:48.210379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.210354    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-856c44d495-6tbh5"]
Jul 08 16:35:48.213504 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.213485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.215668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.215639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g58qt\""
Jul 08 16:35:48.215793 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.215709    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:35:48.215899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.215874    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:35:48.216198 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.216180    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:35:48.216310 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.216292    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:35:48.216374 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.216339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:35:48.216435 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.216407    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:35:48.216484 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.216472    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:35:48.222442 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.222286    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:35:48.226831 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.226811    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856c44d495-6tbh5"]
Jul 08 16:35:48.227412 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.227321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.297032 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.296989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-trusted-ca-bundle\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.297168 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.297041    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-oauth-config\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.297168 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.297093    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-service-ca\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.297295 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.297160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-config\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.297295 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.297212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlzp\" (UniqueName: \"kubernetes.io/projected/4dc56756-9461-4c0d-9442-ebafad2c7c23-kube-api-access-jhlzp\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.297295 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.297245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-serving-cert\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.297423 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.297307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-oauth-serving-cert\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.301942 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.301896    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.397690 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-config\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.397851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlzp\" (UniqueName: \"kubernetes.io/projected/4dc56756-9461-4c0d-9442-ebafad2c7c23-kube-api-access-jhlzp\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.397851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-serving-cert\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.397851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-oauth-serving-cert\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.397851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-trusted-ca-bundle\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.398065 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-oauth-config\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.398065 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.397904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-service-ca\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.398534 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.398506    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-config\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.398638 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.398561    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-service-ca\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.398638 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.398614    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-oauth-serving-cert\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.399310 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.399285    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-trusted-ca-bundle\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.401655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.401614    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-oauth-config\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.401655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.401614    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-serving-cert\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.409435 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.409380    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlzp\" (UniqueName: \"kubernetes.io/projected/4dc56756-9461-4c0d-9442-ebafad2c7c23-kube-api-access-jhlzp\") pod \"console-856c44d495-6tbh5\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") " pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.524922 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.524888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:48.682804 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.682711    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856c44d495-6tbh5"]
Jul 08 16:35:48.687078 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:48.687053    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc56756_9461_4c0d_9442_ebafad2c7c23.slice/crio-d11a6fac6f0e09d20944fbbb28a6e21fe9740f267077d94ea0dab742a33143ed WatchSource:0}: Error finding container d11a6fac6f0e09d20944fbbb28a6e21fe9740f267077d94ea0dab742a33143ed: Status 404 returned error can't find the container with id d11a6fac6f0e09d20944fbbb28a6e21fe9740f267077d94ea0dab742a33143ed
Jul 08 16:35:48.788991 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.788949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.944328 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:48.944226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.161133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:49.161098    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c44d495-6tbh5" event={"ID":"4dc56756-9461-4c0d-9442-ebafad2c7c23","Type":"ContainerStarted","Data":"d11a6fac6f0e09d20944fbbb28a6e21fe9740f267077d94ea0dab742a33143ed"}
Jul 08 16:35:49.647696 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:49.647647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.781419 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:49.781366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.002891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:52.002855    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.107454 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:52.107389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.276767 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:52.276670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.385092 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:52.385052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:54.145655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:54.145621    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hlm6j"
Jul 08 16:35:54.181916 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:54.181873    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c44d495-6tbh5" event={"ID":"4dc56756-9461-4c0d-9442-ebafad2c7c23","Type":"ContainerStarted","Data":"65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6"}
Jul 08 16:35:54.183673 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:54.183651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h242d" event={"ID":"73409722-b9d0-4343-8331-9bc7e60d5f2e","Type":"ContainerStarted","Data":"a60522fcf0b14da95a9ef0f8d985679af3d7eebbfa20f4acc11870f61e20e60c"}
Jul 08 16:35:54.183776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:54.183709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h242d" event={"ID":"73409722-b9d0-4343-8331-9bc7e60d5f2e","Type":"ContainerStarted","Data":"9f47508ddc494c12f176d904bf6421ff258029141f0f925e428dc115541a7ed2"}
Jul 08 16:35:54.202747 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:54.202639    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-856c44d495-6tbh5" podStartSLOduration=1.577138205 podStartE2EDuration="6.202623921s" totalImagesPullingTime="4.625485716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:54.201712267 +0000 UTC m=+124.091127288" watchObservedRunningTime="2026-07-08 16:35:54.202623921 +0000 UTC m=+124.092038965"
Jul 08 16:35:57.196488 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:57.196454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h242d" event={"ID":"73409722-b9d0-4343-8331-9bc7e60d5f2e","Type":"ContainerStarted","Data":"4a6f068a302e9008c1f50af0729851ceb737d2321d618b4a03b0ac4d8ac6c502"}
Jul 08 16:35:57.222900 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:57.219915    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h242d" podStartSLOduration=33.443786012 podStartE2EDuration="42.219896826s" totalImagesPullingTime="8.776110814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:57.218503615 +0000 UTC m=+127.107918625" watchObservedRunningTime="2026-07-08 16:35:57.219896826 +0000 UTC m=+127.109311837"
Jul 08 16:35:58.063389 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.063361    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68447587c9-pfqnx"]
Jul 08 16:35:58.068502 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.068479    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:35:58.147394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.147368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:58.256097 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.256066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:58.525872 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.525845    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:58.526008 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.525896    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:58.530705 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:58.530672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:59.206434 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.206407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:35:59.411160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.411128    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:59.495283 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.495256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:35:59.497430 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.497400    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c52f87-ae67-40cb-8711-1a37de967be6-metrics-certs\") pod \"network-metrics-daemon-txdt2\" (UID: \"33c52f87-ae67-40cb-8711-1a37de967be6\") " pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:35:59.581408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.581384    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwzcp\""
Jul 08 16:35:59.590079 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.590065    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-txdt2"
Jul 08 16:35:59.630527 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.630496    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:59.720876 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.720850    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-txdt2"]
Jul 08 16:35:59.723786 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:35:59.723757    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c52f87_ae67_40cb_8711_1a37de967be6.slice/crio-7fe2a80f00b6ca87b6f83c07fa2e7306e3d8f1ee2a9c8e5b04f4ba591fb61146 WatchSource:0}: Error finding container 7fe2a80f00b6ca87b6f83c07fa2e7306e3d8f1ee2a9c8e5b04f4ba591fb61146: Status 404 returned error can't find the container with id 7fe2a80f00b6ca87b6f83c07fa2e7306e3d8f1ee2a9c8e5b04f4ba591fb61146
Jul 08 16:35:59.726115 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:35:59.726088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:00.207651 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:00.207614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-txdt2" event={"ID":"33c52f87-ae67-40cb-8711-1a37de967be6","Type":"ContainerStarted","Data":"7fe2a80f00b6ca87b6f83c07fa2e7306e3d8f1ee2a9c8e5b04f4ba591fb61146"}
Jul 08 16:36:01.212644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:01.212609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-txdt2" event={"ID":"33c52f87-ae67-40cb-8711-1a37de967be6","Type":"ContainerStarted","Data":"3e38eda81232ba649320f918f3b5be36a623f84ab238a5a199911b43b0ae6ac1"}
Jul 08 16:36:01.213124 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:01.212719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-txdt2" event={"ID":"33c52f87-ae67-40cb-8711-1a37de967be6","Type":"ContainerStarted","Data":"f899e790ec156944a5ed6847803ca23796d650c9db11664e9300ba29042e6ca8"}
Jul 08 16:36:01.230765 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:01.230714    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-txdt2" podStartSLOduration=129.42710751 podStartE2EDuration="2m10.230699342s" totalImagesPullingTime="803.591832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:01.230652426 +0000 UTC m=+131.120067436" watchObservedRunningTime="2026-07-08 16:36:01.230699342 +0000 UTC m=+131.120114344"
Jul 08 16:36:01.335450 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:01.335413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:01.435995 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:01.435960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:01.512779 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:01.512728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.277144 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.277114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.383138 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.383092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.484752 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.484720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.590464 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.590390    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ffksz"]
Jul 08 16:36:02.590820 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.590793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.594085 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.594060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.596739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.596714    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:36:02.597992 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.597976    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:36:02.598143 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.598118    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bnbv8\""
Jul 08 16:36:02.598270 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.598206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:36:02.598270 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.598241    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:36:02.598384 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.598323    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:36:02.598460 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.598441    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:36:02.680452 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.680426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.717367 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-sys\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717480 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-root\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717480 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-textfile\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717480 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92402-a218-4be9-b401-f24ffbefb85c-metrics-client-ca\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717480 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-wtmp\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717711 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fwc\" (UniqueName: \"kubernetes.io/projected/8ef92402-a218-4be9-b401-f24ffbefb85c-kube-api-access-t5fwc\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717711 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717711 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.717711 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.717602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-tls\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.781549 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.781526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.818443 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92402-a218-4be9-b401-f24ffbefb85c-metrics-client-ca\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818542 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-wtmp\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818596 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fwc\" (UniqueName: \"kubernetes.io/projected/8ef92402-a218-4be9-b401-f24ffbefb85c-kube-api-access-t5fwc\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818606    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-wtmp\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-tls\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-sys\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818758    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-root\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818971 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-textfile\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.818971 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818938    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-sys\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.819076 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.818997    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ef92402-a218-4be9-b401-f24ffbefb85c-root\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.819076 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.819054    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92402-a218-4be9-b401-f24ffbefb85c-metrics-client-ca\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.819183 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.819124    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-textfile\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.819231 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.819182    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.821034 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.821013    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.821121 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.821073    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ef92402-a218-4be9-b401-f24ffbefb85c-node-exporter-tls\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.828574 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.828547    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fwc\" (UniqueName: \"kubernetes.io/projected/8ef92402-a218-4be9-b401-f24ffbefb85c-kube-api-access-t5fwc\") pod \"node-exporter-ffksz\" (UID: \"8ef92402-a218-4be9-b401-f24ffbefb85c\") " pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.883942 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.883885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:02.903511 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.903489    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffksz"
Jul 08 16:36:02.915465 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:36:02.915444    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef92402_a218_4be9_b401_f24ffbefb85c.slice/crio-4f95bb23f505b7689a6dcc28391626af322f5612f15156e42e110edb688edef7 WatchSource:0}: Error finding container 4f95bb23f505b7689a6dcc28391626af322f5612f15156e42e110edb688edef7: Status 404 returned error can't find the container with id 4f95bb23f505b7689a6dcc28391626af322f5612f15156e42e110edb688edef7
Jul 08 16:36:02.981228 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:02.981196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.090202 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.090172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.181053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.180962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.219398 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.219369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffksz" event={"ID":"8ef92402-a218-4be9-b401-f24ffbefb85c","Type":"ContainerStarted","Data":"4f95bb23f505b7689a6dcc28391626af322f5612f15156e42e110edb688edef7"}
Jul 08 16:36:03.321718 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.321672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.420538 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.420504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.538583 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.538537    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.636257 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.636219    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:03.930080 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:03.930002    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:04.224396 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.224329    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ef92402-a218-4be9-b401-f24ffbefb85c" containerID="e4a666c353f93ca1cc844f6928b3e7cb3287fdc2cbb460a7b84281453a187761" exitCode=0
Jul 08 16:36:04.224507 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.224421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffksz" event={"ID":"8ef92402-a218-4be9-b401-f24ffbefb85c","Type":"ContainerDied","Data":"e4a666c353f93ca1cc844f6928b3e7cb3287fdc2cbb460a7b84281453a187761"}
Jul 08 16:36:04.544015 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.543987    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"]
Jul 08 16:36:04.547511 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.547490    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.549537 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549515    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 08 16:36:04.549635 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549519    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tmn25\""
Jul 08 16:36:04.549732 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549716    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 08 16:36:04.549833 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549818    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 08 16:36:04.549886 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 08 16:36:04.549886 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 08 16:36:04.549886 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.549871    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-br73pghcjdq1f\""
Jul 08 16:36:04.560881 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.560860    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"]
Jul 08 16:36:04.732306 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-metrics-client-ca\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732315    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-grpc-tls\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdm8\" (UniqueName: \"kubernetes.io/projected/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-kube-api-access-hgdm8\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.732672 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.732489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-tls\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833601 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-metrics-client-ca\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833601 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833601 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-grpc-tls\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdm8\" (UniqueName: \"kubernetes.io/projected/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-kube-api-access-hgdm8\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.833902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.833741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-tls\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.834488 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.834461    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-metrics-client-ca\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.836100 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.836075    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.836358 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.836327    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-tls\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.836564 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.836544    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.836650 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.836627    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.836710 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.836648    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.836785 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.836769    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-secret-grpc-tls\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.845035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.845018    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdm8\" (UniqueName: \"kubernetes.io/projected/d5f69410-a14a-4919-bfb2-f7d54e7d50f2-kube-api-access-hgdm8\") pod \"thanos-querier-77f75d6b49-kbb8r\" (UID: \"d5f69410-a14a-4919-bfb2-f7d54e7d50f2\") " pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.856798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.856780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:04.996945 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:04.996914    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"]
Jul 08 16:36:04.999076 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:36:04.999048    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f69410_a14a_4919_bfb2_f7d54e7d50f2.slice/crio-5d41e845348dcf8b618ee40b1fa7bd04dac7ad09c483d1d5e2277ddb3a832324 WatchSource:0}: Error finding container 5d41e845348dcf8b618ee40b1fa7bd04dac7ad09c483d1d5e2277ddb3a832324: Status 404 returned error can't find the container with id 5d41e845348dcf8b618ee40b1fa7bd04dac7ad09c483d1d5e2277ddb3a832324
Jul 08 16:36:05.229637 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.229607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffksz" event={"ID":"8ef92402-a218-4be9-b401-f24ffbefb85c","Type":"ContainerStarted","Data":"6fe3f2f3199ab3378206365e962be4e5694ada817e2278f95e7559f57db96c7e"}
Jul 08 16:36:05.229809 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.229667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffksz" event={"ID":"8ef92402-a218-4be9-b401-f24ffbefb85c","Type":"ContainerStarted","Data":"c3e2458fdc4a8ab97cc759898dbca28494124751e4b08f5c61ed689fe5baf5fe"}
Jul 08 16:36:05.230751 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.230732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"5d41e845348dcf8b618ee40b1fa7bd04dac7ad09c483d1d5e2277ddb3a832324"}
Jul 08 16:36:05.253967 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.253928    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ffksz" podStartSLOduration=2.085539568 podStartE2EDuration="3.253915436s" totalImagesPullingTime="1.106747925s" totalInitContainerRuntime="61.627943ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:05.252343555 +0000 UTC m=+135.141758566" watchObservedRunningTime="2026-07-08 16:36:05.253915436 +0000 UTC m=+135.143330445"
Jul 08 16:36:05.515891 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.515809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:05.585639 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.585600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:05.825090 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.825011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:05.920322 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:05.920275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.240819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.240785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"01b4dc3d9e4c19052ff89643adebe439599b713d1dc510dab335d7cbb5dee146"}
Jul 08 16:36:07.241160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.240852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"45714a67720436fbe8abfcba7d9018a763c0f946a7c6ebf1594ac2e97f791048"}
Jul 08 16:36:07.241160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.240880    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"82378a01a09ff0dcf38151e46ec96f5057c6b3e49fe1bb01fe85562ccf5bf10b"}
Jul 08 16:36:07.341194 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.341153    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.348923 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.348893    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"]
Jul 08 16:36:07.351998 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.351976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.354525 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.354502    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 08 16:36:07.354614 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.354505    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p2fjg\""
Jul 08 16:36:07.367948 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.367926    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"]
Jul 08 16:36:07.433817 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.433782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.454995 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.454967    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e8cf99a-08eb-48a4-810c-766d6066a86a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-66xjf\" (UID: \"3e8cf99a-08eb-48a4-810c-766d6066a86a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.455129 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.455003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8sb\" (UniqueName: \"kubernetes.io/projected/3e8cf99a-08eb-48a4-810c-766d6066a86a-kube-api-access-lb8sb\") pod \"monitoring-plugin-d5dcff7cf-66xjf\" (UID: \"3e8cf99a-08eb-48a4-810c-766d6066a86a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.533659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.533574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.555350 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.555330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e8cf99a-08eb-48a4-810c-766d6066a86a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-66xjf\" (UID: \"3e8cf99a-08eb-48a4-810c-766d6066a86a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.555455 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.555366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8sb\" (UniqueName: \"kubernetes.io/projected/3e8cf99a-08eb-48a4-810c-766d6066a86a-kube-api-access-lb8sb\") pod \"monitoring-plugin-d5dcff7cf-66xjf\" (UID: \"3e8cf99a-08eb-48a4-810c-766d6066a86a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.557643 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.557622    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3e8cf99a-08eb-48a4-810c-766d6066a86a-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-66xjf\" (UID: \"3e8cf99a-08eb-48a4-810c-766d6066a86a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.565347 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.565325    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8sb\" (UniqueName: \"kubernetes.io/projected/3e8cf99a-08eb-48a4-810c-766d6066a86a-kube-api-access-lb8sb\") pod \"monitoring-plugin-d5dcff7cf-66xjf\" (UID: \"3e8cf99a-08eb-48a4-810c-766d6066a86a\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.634234 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.634176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.662535 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.662514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:07.796346 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:07.796279    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"]
Jul 08 16:36:07.798965 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:36:07.798939    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8cf99a_08eb_48a4_810c_766d6066a86a.slice/crio-b094687ce649f8e9ec165a8d977c7edd07b959a8d4ecbb1e8c1a125e109319f3 WatchSource:0}: Error finding container b094687ce649f8e9ec165a8d977c7edd07b959a8d4ecbb1e8c1a125e109319f3: Status 404 returned error can't find the container with id b094687ce649f8e9ec165a8d977c7edd07b959a8d4ecbb1e8c1a125e109319f3
Jul 08 16:36:08.247096 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.247054    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf" event={"ID":"3e8cf99a-08eb-48a4-810c-766d6066a86a","Type":"ContainerStarted","Data":"b094687ce649f8e9ec165a8d977c7edd07b959a8d4ecbb1e8c1a125e109319f3"}
Jul 08 16:36:08.250509 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.250472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"28b9f684e42b68d77abc9720785c3c744d4f37f942b2f318ae398dc0d2f4cf68"}
Jul 08 16:36:08.250644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.250547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"6b805888be6c066f0bb2c5ea6fb6775ed5c89a3996d8211de30017ff877ee4d1"}
Jul 08 16:36:08.250644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.250590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" event={"ID":"d5f69410-a14a-4919-bfb2-f7d54e7d50f2","Type":"ContainerStarted","Data":"7eaf0e8219bfa11b718441571334cd6191c0e9cd48104e00a129fd7923ff6d76"}
Jul 08 16:36:08.250789 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.250646    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:08.276698 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.276643    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r" podStartSLOduration=1.8855117639999999 podStartE2EDuration="4.276630246s" totalImagesPullingTime="2.391118482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:08.27521406 +0000 UTC m=+138.164629070" watchObservedRunningTime="2026-07-08 16:36:08.276630246 +0000 UTC m=+138.166045259"
Jul 08 16:36:08.808160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.808069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:08.809078 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.809050    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:36:08.814568 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.814544    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.820255 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820050    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 16:36:08.820349 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820287    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 16:36:08.820421 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820365    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 16:36:08.820554 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820537    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 16:36:08.820619 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820593    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 16:36:08.820673 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820620    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 16:36:08.820748 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820701    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 16:36:08.820796 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820784    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 16:36:08.820905 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820886    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3esmj2pd8gj3l\""
Jul 08 16:36:08.820965 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820909    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 16:36:08.820965 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.820921    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-26fll\""
Jul 08 16:36:08.821074 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.821056    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 16:36:08.821131 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.821088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 16:36:08.821131 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.821110    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 16:36:08.821233 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.821135    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 16:36:08.821233 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.821090    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 16:36:08.823938 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.823774    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 16:36:08.840557 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.840535    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:36:08.864846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.864820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.864959 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.864859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.864959 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.864876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.864959 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.864937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865151 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.864972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865151 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865151 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865151 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865151 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kfn\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-kube-api-access-d2kfn\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-config\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.865609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.865556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.965851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.965826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.965966 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.965865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.965966 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.965895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kfn\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-kube-api-access-d2kfn\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.965966 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.965930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.965966 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.965954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.965983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-config\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966216    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966483 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966944 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966549    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.966944 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.966785    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.967224 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.967184    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.967962 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.967774    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.968476 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.968444    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.968567 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.968453    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.969457 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.969370    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-web-config\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.969568 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.969545    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.969749 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.969729    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.969830 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.969729    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.969907 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.969883    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.969993 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.969967    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-config-out\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.970988 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.970967    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.971367 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.971337    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.971459 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.971439    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.971670 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.971649    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-config\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.971872 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.971855    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.971915 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.971899    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:08.973692 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:08.973663    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kfn\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-kube-api-access-d2kfn\") pod \"prometheus-k8s-0\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:09.127334 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:09.127258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:09.257424 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:09.257393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf" event={"ID":"3e8cf99a-08eb-48a4-810c-766d6066a86a","Type":"ContainerStarted","Data":"05e560ac08c2a520653674b004515dfd519a7fb11bc7408b15bd8b0f9e359682"}
Jul 08 16:36:09.257874 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:09.257718    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:09.262135 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:09.262105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf"
Jul 08 16:36:09.265010 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:09.264975    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:36:09.266727 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:36:09.266705    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cff45e_af55_4939_ae94_e62661edf8f3.slice/crio-065695e5386bbff2e29987319775c1447ab7918d0a1e4422974ed8d579e13560 WatchSource:0}: Error finding container 065695e5386bbff2e29987319775c1447ab7918d0a1e4422974ed8d579e13560: Status 404 returned error can't find the container with id 065695e5386bbff2e29987319775c1447ab7918d0a1e4422974ed8d579e13560
Jul 08 16:36:09.277240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:09.277202    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-66xjf" podStartSLOduration=1.141370075 podStartE2EDuration="2.277188145s" totalImagesPullingTime="1.13581807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:09.276657558 +0000 UTC m=+139.166072563" watchObservedRunningTime="2026-07-08 16:36:09.277188145 +0000 UTC m=+139.166603154"
Jul 08 16:36:10.074809 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:10.074773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:10.178674 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:10.178635    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:10.262669 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:10.262646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"065695e5386bbff2e29987319775c1447ab7918d0a1e4422974ed8d579e13560"}
Jul 08 16:36:11.267226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:11.267191    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b" exitCode=0
Jul 08 16:36:11.267567 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:11.267272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"}
Jul 08 16:36:12.296033 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:12.296000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:12.397050 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:12.397016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:13.351459 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:13.351414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:13.410873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:13.410836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:13.948755 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:13.948713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:14.013533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:14.013491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:14.265579 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:14.265541    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77f75d6b49-kbb8r"
Jul 08 16:36:15.285449 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.285418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"}
Jul 08 16:36:15.285846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.285483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"}
Jul 08 16:36:15.285846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.285510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"}
Jul 08 16:36:15.285846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.285536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"}
Jul 08 16:36:15.285846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.285560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"}
Jul 08 16:36:15.285846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.285585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerStarted","Data":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"}
Jul 08 16:36:15.312899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:15.312854    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.10319445 podStartE2EDuration="7.3128387s" totalImagesPullingTime="4.190554016s" totalInitContainerRuntime="19.090234ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:15.311042971 +0000 UTC m=+145.200457978" watchObservedRunningTime="2026-07-08 16:36:15.3128387 +0000 UTC m=+145.202253710"
Jul 08 16:36:18.897553 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:18.897512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:18.992905 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:18.992874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:19.128361 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:19.128331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:23.082988 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.082950    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68447587c9-pfqnx" podUID="c839748c-70cd-4055-972b-f9c232a0b040" containerName="registry" containerID="cri-o://39f0397ce369b4e02081b6b787111a379fd1f50defc80a1eb5b3cef026b6bfb3" gracePeriod=30
Jul 08 16:36:23.316560 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.316527    2559 generic.go:356] "Generic (PLEG): container finished" podID="c839748c-70cd-4055-972b-f9c232a0b040" containerID="39f0397ce369b4e02081b6b787111a379fd1f50defc80a1eb5b3cef026b6bfb3" exitCode=0
Jul 08 16:36:23.316701 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.316597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68447587c9-pfqnx" event={"ID":"c839748c-70cd-4055-972b-f9c232a0b040","Type":"ContainerDied","Data":"39f0397ce369b4e02081b6b787111a379fd1f50defc80a1eb5b3cef026b6bfb3"}
Jul 08 16:36:23.873748 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.873723    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:36:23.899100 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.899072    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856c44d495-6tbh5"]
Jul 08 16:36:23.986093 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-registry-certificates\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986247 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-image-registry-private-configuration\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986247 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986135    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-installation-pull-secrets\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986247 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986222    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986413 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986273    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-trusted-ca\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986413 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986303    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c839748c-70cd-4055-972b-f9c232a0b040-ca-trust-extracted\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986413 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986340    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqwz9\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-kube-api-access-wqwz9\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986413 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986367    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-bound-sa-token\") pod \"c839748c-70cd-4055-972b-f9c232a0b040\" (UID: \"c839748c-70cd-4055-972b-f9c232a0b040\") "
Jul 08 16:36:23.986620 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986478    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:23.986790 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986766    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-registry-certificates\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:23.986873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.986785    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:23.988698 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.988649    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:23.988920 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.988892    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:23.989025 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.988973    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:23.989163 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.989143    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-kube-api-access-wqwz9" (OuterVolumeSpecName: "kube-api-access-wqwz9") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "kube-api-access-wqwz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:23.989216 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.989200    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:23.994526 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:23.994505    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c839748c-70cd-4055-972b-f9c232a0b040-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c839748c-70cd-4055-972b-f9c232a0b040" (UID: "c839748c-70cd-4055-972b-f9c232a0b040"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:36:24.013589 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.013562    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:24.087226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087204    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-image-registry-private-configuration\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.087226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087226    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c839748c-70cd-4055-972b-f9c232a0b040-installation-pull-secrets\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.087562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087236    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-registry-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.087562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087245    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c839748c-70cd-4055-972b-f9c232a0b040-trusted-ca\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.087562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087253    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c839748c-70cd-4055-972b-f9c232a0b040-ca-trust-extracted\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.087562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087261    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwz9\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-kube-api-access-wqwz9\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.087562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.087269    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c839748c-70cd-4055-972b-f9c232a0b040-bound-sa-token\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:24.113438 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.113408    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:24.179752 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.179732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:24.321373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.321304    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68447587c9-pfqnx"
Jul 08 16:36:24.321454 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.321304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68447587c9-pfqnx" event={"ID":"c839748c-70cd-4055-972b-f9c232a0b040","Type":"ContainerDied","Data":"5514d43ed68aa5a3db61123f5619fcbce8e98ebdc4aa1d4c977df5e904cfc856"}
Jul 08 16:36:24.321454 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.321436    2559 scope.go:117] "RemoveContainer" containerID="39f0397ce369b4e02081b6b787111a379fd1f50defc80a1eb5b3cef026b6bfb3"
Jul 08 16:36:24.343543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.343520    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68447587c9-pfqnx"]
Jul 08 16:36:24.348278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.348256    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68447587c9-pfqnx"]
Jul 08 16:36:24.672783 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:24.672723    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c839748c-70cd-4055-972b-f9c232a0b040" path="/var/lib/kubelet/pods/c839748c-70cd-4055-972b-f9c232a0b040/volumes"
Jul 08 16:36:25.070152 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:25.070114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:25.130698 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:25.130653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:28.982078 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:28.982043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:29.050806 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:29.050779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:33.412639 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:33.412602    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:33.483944 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:33.483913    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:48.919800 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:48.919721    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-856c44d495-6tbh5" podUID="4dc56756-9461-4c0d-9442-ebafad2c7c23" containerName="console" containerID="cri-o://65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6" gracePeriod=15
Jul 08 16:36:49.175142 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.175084    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:36:49.272376 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272351    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-oauth-config\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272385    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-oauth-serving-cert\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272404    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-service-ca\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhlzp\" (UniqueName: \"kubernetes.io/projected/4dc56756-9461-4c0d-9442-ebafad2c7c23-kube-api-access-jhlzp\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272716 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272618    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-config\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272716 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272694    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-trusted-ca-bundle\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272716 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272713    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-serving-cert\") pod \"4dc56756-9461-4c0d-9442-ebafad2c7c23\" (UID: \"4dc56756-9461-4c0d-9442-ebafad2c7c23\") "
Jul 08 16:36:49.272937 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272818    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-service-ca" (OuterVolumeSpecName: "service-ca") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:49.273023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.272937    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:49.273023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.273001    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-config" (OuterVolumeSpecName: "console-config") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:49.273130 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.273079    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:49.274617 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.274598    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:49.274698 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.274632    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc56756-9461-4c0d-9442-ebafad2c7c23-kube-api-access-jhlzp" (OuterVolumeSpecName: "kube-api-access-jhlzp") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "kube-api-access-jhlzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:49.274862 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.274843    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4dc56756-9461-4c0d-9442-ebafad2c7c23" (UID: "4dc56756-9461-4c0d-9442-ebafad2c7c23"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:49.373727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373700    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhlzp\" (UniqueName: \"kubernetes.io/projected/4dc56756-9461-4c0d-9442-ebafad2c7c23-kube-api-access-jhlzp\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.373727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373724    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.373857 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373739    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-trusted-ca-bundle\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.373857 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373748    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-serving-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.373857 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373756    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4dc56756-9461-4c0d-9442-ebafad2c7c23-console-oauth-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.373857 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373763    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-oauth-serving-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.373857 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.373771    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc56756-9461-4c0d-9442-ebafad2c7c23-service-ca\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:36:49.413947 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.413919    2559 generic.go:356] "Generic (PLEG): container finished" podID="4dc56756-9461-4c0d-9442-ebafad2c7c23" containerID="65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6" exitCode=0
Jul 08 16:36:49.414036 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.413968    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856c44d495-6tbh5"
Jul 08 16:36:49.414036 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.414006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c44d495-6tbh5" event={"ID":"4dc56756-9461-4c0d-9442-ebafad2c7c23","Type":"ContainerDied","Data":"65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6"}
Jul 08 16:36:49.414117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.414059    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856c44d495-6tbh5" event={"ID":"4dc56756-9461-4c0d-9442-ebafad2c7c23","Type":"ContainerDied","Data":"d11a6fac6f0e09d20944fbbb28a6e21fe9740f267077d94ea0dab742a33143ed"}
Jul 08 16:36:49.414117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.414088    2559 scope.go:117] "RemoveContainer" containerID="65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6"
Jul 08 16:36:49.426792 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.426773    2559 scope.go:117] "RemoveContainer" containerID="65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6"
Jul 08 16:36:49.427089 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:36:49.427069    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6\": container with ID starting with 65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6 not found: ID does not exist" containerID="65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6"
Jul 08 16:36:49.427155 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.427102    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6"} err="failed to get container status \"65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6\": rpc error: code = NotFound desc = could not find container \"65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6\": container with ID starting with 65ce02885e844de7665e0353fd5bef715e068554c62505de31e4a50765e5b2e6 not found: ID does not exist"
Jul 08 16:36:49.438373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.438341    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856c44d495-6tbh5"]
Jul 08 16:36:49.442923 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:49.442903    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-856c44d495-6tbh5"]
Jul 08 16:36:50.673137 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:36:50.673108    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc56756-9461-4c0d-9442-ebafad2c7c23" path="/var/lib/kubelet/pods/4dc56756-9461-4c0d-9442-ebafad2c7c23/volumes"
Jul 08 16:37:09.128326 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:09.128295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:09.146240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:09.146220    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:09.506427 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:09.506403    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:17.568710 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:17.568636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:18.703727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:18.703663    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:18.767935 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:18.767905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:27.278517 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278486    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:37:27.279072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278800    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="prometheus" containerID="cri-o://328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69" gracePeriod=600
Jul 08 16:37:27.279072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278830    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy" containerID="cri-o://7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4" gracePeriod=600
Jul 08 16:37:27.279072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278880    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5" gracePeriod=600
Jul 08 16:37:27.279072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278895    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-web" containerID="cri-o://887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4" gracePeriod=600
Jul 08 16:37:27.279072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278834    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="thanos-sidecar" containerID="cri-o://420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149" gracePeriod=600
Jul 08 16:37:27.279072 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.278888    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="config-reloader" containerID="cri-o://567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca" gracePeriod=600
Jul 08 16:37:27.524299 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.524273    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.537513 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537464    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2kfn\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-kube-api-access-d2kfn\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537513 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537494    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-kube-rbac-proxy\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537526    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-2\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537543    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-web-config\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-trusted-ca-bundle\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537577    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-config-out\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537599    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-tls\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537623    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-thanos-prometheus-http-client-file\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.537664 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-serving-certs-ca-bundle\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537738    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537790    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-tls-assets\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-1\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537866    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-0\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-metrics-client-ca\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537916    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537941    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-grpc-tls\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.537967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-db\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538003    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.538035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-config\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538057    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-metrics-client-certs\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538082    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-kubelet-serving-ca-bundle\") pod \"96cff45e-af55-4939-ae94-e62661edf8f3\" (UID: \"96cff45e-af55-4939-ae94-e62661edf8f3\") "
Jul 08 16:37:27.538572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538239    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.538572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538322    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.538572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538341    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.538572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.538358    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.539486 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.539458    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.540363 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.540340    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:37:27.540447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.540345    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-kube-api-access-d2kfn" (OuterVolumeSpecName: "kube-api-access-d2kfn") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "kube-api-access-d2kfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:37:27.540875 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.540851    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.541264 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.541238    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.542069 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542037    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:27.542167 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542106    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.542167 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542122    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:37:27.542167 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542144    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.542339 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542200    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.542450 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542418    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-config-out" (OuterVolumeSpecName: "config-out") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:37:27.542667 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542645    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.542761 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542669    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.542956 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.542937    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.543060 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.543044    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-config" (OuterVolumeSpecName: "config") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.543142 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.543118    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.551169 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.551149    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-web-config" (OuterVolumeSpecName: "web-config") pod "96cff45e-af55-4939-ae94-e62661edf8f3" (UID: "96cff45e-af55-4939-ae94-e62661edf8f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:27.567626 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567578    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5" exitCode=0
Jul 08 16:37:27.567626 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567612    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4" exitCode=0
Jul 08 16:37:27.567626 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567621    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4" exitCode=0
Jul 08 16:37:27.567626 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567629    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149" exitCode=0
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567637    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca" exitCode=0
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567645    2559 generic.go:356] "Generic (PLEG): container finished" podID="96cff45e-af55-4939-ae94-e62661edf8f3" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69" exitCode=0
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567719    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"}
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"}
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"}
Jul 08 16:37:27.567943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"}
Jul 08 16:37:27.568279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567953    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.568279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"}
Jul 08 16:37:27.568279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.567992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"}
Jul 08 16:37:27.568279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.568015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96cff45e-af55-4939-ae94-e62661edf8f3","Type":"ContainerDied","Data":"065695e5386bbff2e29987319775c1447ab7918d0a1e4422974ed8d579e13560"}
Jul 08 16:37:27.576930 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.576815    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.584312 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.584296    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.590914 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.590893    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.593624 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.593604    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:37:27.598208 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.598189    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.604667 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.604646    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:37:27.605046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.605030    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.613174 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.613156    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.619287 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.619271    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.619530 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.619511    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.619576 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.619537    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"} err="failed to get container status \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist"
Jul 08 16:37:27.619576 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.619553    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.619794 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.619777    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.619868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.619809    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"} err="failed to get container status \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist"
Jul 08 16:37:27.619868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.619831    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.620098 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.620082    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.620134 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.620104    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"} err="failed to get container status \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist"
Jul 08 16:37:27.620134 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.620118    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.620345 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.620329    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.620408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.620352    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"} err="failed to get container status \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist"
Jul 08 16:37:27.620408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.620372    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.620607 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.620592    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.620644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.620612    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"} err="failed to get container status \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist"
Jul 08 16:37:27.620644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.620626    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.621156 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.621140    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.621218 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621163    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"} err="failed to get container status \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist"
Jul 08 16:37:27.621218 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621182    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.621389 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:37:27.621372    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.621448 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621395    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"} err="failed to get container status \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist"
Jul 08 16:37:27.621448 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621414    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.621639 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621621    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"} err="failed to get container status \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist"
Jul 08 16:37:27.621639 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621638    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.621904 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621889    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"} err="failed to get container status \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist"
Jul 08 16:37:27.621904 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.621904    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.622118 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622103    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"} err="failed to get container status \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist"
Jul 08 16:37:27.622118 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622118    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.622330 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622316    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"} err="failed to get container status \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist"
Jul 08 16:37:27.622330 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622330    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.622566 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622546    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"} err="failed to get container status \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist"
Jul 08 16:37:27.622661 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622591    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.622830 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622813    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"} err="failed to get container status \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist"
Jul 08 16:37:27.622879 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.622833    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.623021 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623003    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"} err="failed to get container status \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist"
Jul 08 16:37:27.623081 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623022    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.623235 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623219    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"} err="failed to get container status \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist"
Jul 08 16:37:27.623298 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623236    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.623441 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623426    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"} err="failed to get container status \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist"
Jul 08 16:37:27.623489 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623441    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.623672 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623653    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"} err="failed to get container status \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist"
Jul 08 16:37:27.623747 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623673    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.623904 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623889    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"} err="failed to get container status \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist"
Jul 08 16:37:27.623947 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.623904    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.624095 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624077    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"} err="failed to get container status \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist"
Jul 08 16:37:27.624161 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624097    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.624301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624284    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"} err="failed to get container status \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist"
Jul 08 16:37:27.624345 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624303    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.624465 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624446    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"} err="failed to get container status \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist"
Jul 08 16:37:27.624515 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624466    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.624650 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624636    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"} err="failed to get container status \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist"
Jul 08 16:37:27.624650 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624649    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.624895 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624875    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"} err="failed to get container status \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist"
Jul 08 16:37:27.624949 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.624904    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.625108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625093    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"} err="failed to get container status \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist"
Jul 08 16:37:27.625108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625108    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.625291 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625274    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"} err="failed to get container status \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist"
Jul 08 16:37:27.625291 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625290    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.625462 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625442    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"} err="failed to get container status \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist"
Jul 08 16:37:27.625502 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625463    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.625668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625651    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"} err="failed to get container status \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist"
Jul 08 16:37:27.625796 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625670    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.625917 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625902    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"} err="failed to get container status \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist"
Jul 08 16:37:27.625957 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.625918    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.626121 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626104    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"} err="failed to get container status \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist"
Jul 08 16:37:27.626175 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626121    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.626321 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626303    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"} err="failed to get container status \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist"
Jul 08 16:37:27.626321 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626319    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.626498 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626483    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"} err="failed to get container status \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist"
Jul 08 16:37:27.626542 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626499    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.626668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626652    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"} err="failed to get container status \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist"
Jul 08 16:37:27.626735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626669    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.626878 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626864    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"} err="failed to get container status \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist"
Jul 08 16:37:27.626920 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.626878    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.627046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627031    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"} err="failed to get container status \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist"
Jul 08 16:37:27.627082 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627046    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.627256 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627239    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"} err="failed to get container status \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist"
Jul 08 16:37:27.627256 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627256    2559 scope.go:117] "RemoveContainer" containerID="50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"
Jul 08 16:37:27.627435 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627419    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5"} err="failed to get container status \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": rpc error: code = NotFound desc = could not find container \"50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5\": container with ID starting with 50790862e4cbd97761d1081f44f4f2337d8a87bbeb3bd891b60e08424eff21b5 not found: ID does not exist"
Jul 08 16:37:27.627473 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627435    2559 scope.go:117] "RemoveContainer" containerID="7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"
Jul 08 16:37:27.627604 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627590    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4"} err="failed to get container status \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": rpc error: code = NotFound desc = could not find container \"7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4\": container with ID starting with 7c0d14bc7a770e3df94f6e601a6015e957a75f296dc9aaa07821340e2f51b0e4 not found: ID does not exist"
Jul 08 16:37:27.627604 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627604    2559 scope.go:117] "RemoveContainer" containerID="887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"
Jul 08 16:37:27.627806 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627787    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4"} err="failed to get container status \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": rpc error: code = NotFound desc = could not find container \"887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4\": container with ID starting with 887823a102b07b240ffeb51e9b47ba71dae20230c1bb92d4905d0de7e85393e4 not found: ID does not exist"
Jul 08 16:37:27.627806 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627804    2559 scope.go:117] "RemoveContainer" containerID="420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"
Jul 08 16:37:27.628009 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.627990    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149"} err="failed to get container status \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": rpc error: code = NotFound desc = could not find container \"420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149\": container with ID starting with 420c7e71d4d2ee1aa10f4c10cb98319680ebdd81b7c6cb7fabc1a68eab471149 not found: ID does not exist"
Jul 08 16:37:27.628048 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.628009    2559 scope.go:117] "RemoveContainer" containerID="567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"
Jul 08 16:37:27.628252 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.628238    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca"} err="failed to get container status \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": rpc error: code = NotFound desc = could not find container \"567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca\": container with ID starting with 567f5096cf88315b85528156572a2231a0f9ff56690b6c3401952fd33be431ca not found: ID does not exist"
Jul 08 16:37:27.628290 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.628252    2559 scope.go:117] "RemoveContainer" containerID="328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"
Jul 08 16:37:27.628448 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.628432    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69"} err="failed to get container status \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": rpc error: code = NotFound desc = could not find container \"328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69\": container with ID starting with 328e3c02b04faa3e6237fa318b780195cc33dfacf0365ddb40bb92dff5fdee69 not found: ID does not exist"
Jul 08 16:37:27.628485 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.628448    2559 scope.go:117] "RemoveContainer" containerID="d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"
Jul 08 16:37:27.628647 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.628628    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b"} err="failed to get container status \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": rpc error: code = NotFound desc = could not find container \"d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b\": container with ID starting with d763cb0fbdcf59c179115fcec2d616315b526ef61668421592b950056bd0a39b not found: ID does not exist"
Jul 08 16:37:27.632354 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632328    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:37:27.632573 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632563    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="init-config-reloader"
Jul 08 16:37:27.632613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632575    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="init-config-reloader"
Jul 08 16:37:27.632613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632585    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c839748c-70cd-4055-972b-f9c232a0b040" containerName="registry"
Jul 08 16:37:27.632613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632591    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c839748c-70cd-4055-972b-f9c232a0b040" containerName="registry"
Jul 08 16:37:27.632613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632601    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy"
Jul 08 16:37:27.632613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632606    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy"
Jul 08 16:37:27.632613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632612    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-thanos"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632617    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-thanos"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632624    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="prometheus"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632629    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="prometheus"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632635    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="config-reloader"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632640    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="config-reloader"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632648    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc56756-9461-4c0d-9442-ebafad2c7c23" containerName="console"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632653    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc56756-9461-4c0d-9442-ebafad2c7c23" containerName="console"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632658    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="thanos-sidecar"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632663    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="thanos-sidecar"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632668    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-web"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632673    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-web"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632732    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c839748c-70cd-4055-972b-f9c232a0b040" containerName="registry"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632740    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc56756-9461-4c0d-9442-ebafad2c7c23" containerName="console"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632746    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-web"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632752    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy-thanos"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632758    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="prometheus"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632763    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="thanos-sidecar"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632769    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="config-reloader"
Jul 08 16:37:27.632819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.632774    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" containerName="kube-rbac-proxy"
Jul 08 16:37:27.637834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.637819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.638695 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638657    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-kube-rbac-proxy\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.638777 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638718    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-web-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.638777 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638735    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-config-out\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.638777 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638749    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.638777 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638762    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638776    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638790    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-tls-assets\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638802    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638814    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638828    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-metrics-client-ca\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638841    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-grpc-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638855    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638868    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96cff45e-af55-4939-ae94-e62661edf8f3-prometheus-k8s-db\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638881    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638894    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96cff45e-af55-4939-ae94-e62661edf8f3-secret-metrics-client-certs\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638910    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96cff45e-af55-4939-ae94-e62661edf8f3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.638923    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kfn\" (UniqueName: \"kubernetes.io/projected/96cff45e-af55-4939-ae94-e62661edf8f3-kube-api-access-d2kfn\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:37:27.639887 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.639870    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 16:37:27.640026 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.639988    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3esmj2pd8gj3l\""
Jul 08 16:37:27.640328 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640310    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 16:37:27.640784 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 16:37:27.640784 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640759    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 16:37:27.640946 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640786    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 16:37:27.640946 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640798    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 16:37:27.640946 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 16:37:27.640946 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.640758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 16:37:27.641102 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.641039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 16:37:27.641102 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.641096    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-26fll\""
Jul 08 16:37:27.641592 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.641576    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 16:37:27.641670 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.641631    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 16:37:27.641739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.641635    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 16:37:27.641791 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.641746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 16:37:27.643494 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.643477    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 16:37:27.646562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.646543    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 16:37:27.652343 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.652322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:37:27.739922 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.739896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.739929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.739947    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.739963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740152 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740183 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e37608cc-9544-49fe-8697-a6a91e1282e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740215 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740215 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e37608cc-9544-49fe-8697-a6a91e1282e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssbw\" (UniqueName: \"kubernetes.io/projected/e37608cc-9544-49fe-8697-a6a91e1282e7-kube-api-access-bssbw\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740350    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740560 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740560 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-config\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.740560 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.740467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841384 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841384 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e37608cc-9544-49fe-8697-a6a91e1282e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841384 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841665 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e37608cc-9544-49fe-8697-a6a91e1282e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841665 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841665 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841665 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssbw\" (UniqueName: \"kubernetes.io/projected/e37608cc-9544-49fe-8697-a6a91e1282e7-kube-api-access-bssbw\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-config\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.841889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.841972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.842067    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.842182    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842545 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.842517    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.842821 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.842796    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.844263 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.844145    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e37608cc-9544-49fe-8697-a6a91e1282e7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.844565 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.844543    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.844641 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.844612    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845032 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845008    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e37608cc-9544-49fe-8697-a6a91e1282e7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845111 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845026    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845314    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845402    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845590    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845895 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845670    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.845980 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845956    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.846039 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.845978    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e37608cc-9544-49fe-8697-a6a91e1282e7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.846731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.846712    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.846862 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.846842    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.847116 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.847100    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-config\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.847259 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.847243    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e37608cc-9544-49fe-8697-a6a91e1282e7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.851287 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.851271    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssbw\" (UniqueName: \"kubernetes.io/projected/e37608cc-9544-49fe-8697-a6a91e1282e7-kube-api-access-bssbw\") pod \"prometheus-k8s-0\" (UID: \"e37608cc-9544-49fe-8697-a6a91e1282e7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:27.947974 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:27.947952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:28.080464 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:28.080273    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:37:28.082483 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:37:28.082453    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37608cc_9544_49fe_8697_a6a91e1282e7.slice/crio-d769ee21aff81db16bd1bfdb74e5ee244428858a68fa7e46da870682b7cb4cef WatchSource:0}: Error finding container d769ee21aff81db16bd1bfdb74e5ee244428858a68fa7e46da870682b7cb4cef: Status 404 returned error can't find the container with id d769ee21aff81db16bd1bfdb74e5ee244428858a68fa7e46da870682b7cb4cef
Jul 08 16:37:28.572942 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:28.572908    2559 generic.go:356] "Generic (PLEG): container finished" podID="e37608cc-9544-49fe-8697-a6a91e1282e7" containerID="ff20cd3edd9dfa7ffcbbeb22629d001504c780347f5845f5922f1e80740e96ce" exitCode=0
Jul 08 16:37:28.573366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:28.572993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerDied","Data":"ff20cd3edd9dfa7ffcbbeb22629d001504c780347f5845f5922f1e80740e96ce"}
Jul 08 16:37:28.573366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:28.573080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"d769ee21aff81db16bd1bfdb74e5ee244428858a68fa7e46da870682b7cb4cef"}
Jul 08 16:37:28.674023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:28.673999    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cff45e-af55-4939-ae94-e62661edf8f3" path="/var/lib/kubelet/pods/96cff45e-af55-4939-ae94-e62661edf8f3/volumes"
Jul 08 16:37:29.581775 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.581737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"a3a6381f936fb27a07149aa438cd8cbcbe0d241a83c2e062b01d709129e82186"}
Jul 08 16:37:29.582148 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.581838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"a214378b43015a3d1087d202cdd3e58c9238b69e7b77d4f796079cf7ad5872d5"}
Jul 08 16:37:29.582148 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.581865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"0adbb02eeb11e587cb2e1cc553a5c6f75df78fabe4769478260e0ca51840101f"}
Jul 08 16:37:29.582148 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.581890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"fae7a9a63b9c45d2c0837f93aeade00074e0cb30b8a5166d340d9824bf109171"}
Jul 08 16:37:29.582148 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.581925    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"d945c19b6692313d7dc44e2209f1fb6664a7c4f12287bc553476ef47007741d9"}
Jul 08 16:37:29.582148 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.581951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e37608cc-9544-49fe-8697-a6a91e1282e7","Type":"ContainerStarted","Data":"1273babe66b84b60ae599f19f55f4ea615452a038b53db5577092ca9e02e572f"}
Jul 08 16:37:29.610113 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:29.610064    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5889135469999998 podStartE2EDuration="2.610047545s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.133998ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:29.608591346 +0000 UTC m=+219.498006356" watchObservedRunningTime="2026-07-08 16:37:29.610047545 +0000 UTC m=+219.499462555"
Jul 08 16:37:30.596399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:30.596368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:30.653335 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:30.653308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:32.182160 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:32.182123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:32.282498 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:32.282464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:32.948622 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:32.948591    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:37:38.544637 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:38.544600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:40.082252 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:40.082218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:40.182852 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:40.182812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:40.248792 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:40.248759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:43.300540 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:43.300511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:43.359779 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:43.359740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:43.420970 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:37:43.420949    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:02.951330 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:02.951297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:16.382482 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:16.382395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:27.948919 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:27.948888    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:38:27.962815 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:27.962793    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:38:28.805120 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:28.805096    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:38:50.543287 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:50.543255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:38:50.545814 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:50.545794    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:38:50.550578 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:50.550558    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:38:57.992092 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:57.992061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:58.054621 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:58.054591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:58.411052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:58.410967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:58.509391 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:58.509358    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:58.570111 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:38:58.570087    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:21.647349 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:21.647314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:21.813982 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:21.813956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:29.047404 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:29.047363    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:29.435595 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:29.435511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:29.734958 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:29.734906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:29.967698 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:29.967647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:30.191393 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:30.191305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:30.495093 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:30.495058    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:30.606820 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:30.606787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:31.113348 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:31.113304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:31.362233 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:31.362197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:31.515418 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:31.515376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:46.412265 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:39:46.412184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:33.785742 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:33.785701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:33.885976 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:33.885923    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:33.961572 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:33.961532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:34.046716 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:34.046627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:34.711462 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:34.711426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:34.810645 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:34.810613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:34.871609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:40:34.871578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:16.501390 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:16.501314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:27.155046 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:27.155014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:27.266902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:27.266842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:27.367599 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:27.367574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:43.971171 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:43.971136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:44.049738 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:44.049705    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:44.271674 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:44.271640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:44.366320 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:44.366290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:44.734744 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:44.734719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:44.789232 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:44.789208    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:59.952890 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.952855    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-f8gsq"]
Jul 08 16:41:59.956077 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.956061    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f8gsq"
Jul 08 16:41:59.957921 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.957900    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 16:41:59.959215 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.959193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cjbv7\""
Jul 08 16:41:59.959309 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.959212    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:41:59.959428 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.959416    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:41:59.963747 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:41:59.963724    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-f8gsq"]
Jul 08 16:42:00.042421 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.042393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c54020-3aeb-47f0-9395-faebb4a13865-model-cache\") pod \"s3-init-f8gsq\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") " pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.042519 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.042429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgqr\" (UniqueName: \"kubernetes.io/projected/52c54020-3aeb-47f0-9395-faebb4a13865-kube-api-access-9sgqr\") pod \"s3-init-f8gsq\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") " pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.142774 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.142751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c54020-3aeb-47f0-9395-faebb4a13865-model-cache\") pod \"s3-init-f8gsq\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") " pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.142884 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.142776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgqr\" (UniqueName: \"kubernetes.io/projected/52c54020-3aeb-47f0-9395-faebb4a13865-kube-api-access-9sgqr\") pod \"s3-init-f8gsq\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") " pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.143126 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.143106    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c54020-3aeb-47f0-9395-faebb4a13865-model-cache\") pod \"s3-init-f8gsq\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") " pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.161847 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.161825    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgqr\" (UniqueName: \"kubernetes.io/projected/52c54020-3aeb-47f0-9395-faebb4a13865-kube-api-access-9sgqr\") pod \"s3-init-f8gsq\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") " pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.265440 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.265418    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f8gsq"
Jul 08 16:42:00.394122 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.394096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-f8gsq"]
Jul 08 16:42:00.396635 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:42:00.396610    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52c54020_3aeb_47f0_9395_faebb4a13865.slice/crio-ecbc2a27fa8599b99ef4a56882facdccac92b3aa4ec43a3d6bb0c2fc5dc8c806 WatchSource:0}: Error finding container ecbc2a27fa8599b99ef4a56882facdccac92b3aa4ec43a3d6bb0c2fc5dc8c806: Status 404 returned error can't find the container with id ecbc2a27fa8599b99ef4a56882facdccac92b3aa4ec43a3d6bb0c2fc5dc8c806
Jul 08 16:42:00.398921 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.398897    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:42:00.545298 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:00.545237    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f8gsq" event={"ID":"52c54020-3aeb-47f0-9395-faebb4a13865","Type":"ContainerStarted","Data":"ecbc2a27fa8599b99ef4a56882facdccac92b3aa4ec43a3d6bb0c2fc5dc8c806"}
Jul 08 16:42:05.569329 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:05.569295    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f8gsq" event={"ID":"52c54020-3aeb-47f0-9395-faebb4a13865","Type":"ContainerStarted","Data":"a31f4c40cbcc0320f70781066862ccf88387fd54c7f51d399d7376d847ed49e1"}
Jul 08 16:42:07.578431 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:07.578398    2559 generic.go:356] "Generic (PLEG): container finished" podID="52c54020-3aeb-47f0-9395-faebb4a13865" containerID="a31f4c40cbcc0320f70781066862ccf88387fd54c7f51d399d7376d847ed49e1" exitCode=0
Jul 08 16:42:07.578833 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:07.578472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f8gsq" event={"ID":"52c54020-3aeb-47f0-9395-faebb4a13865","Type":"ContainerDied","Data":"a31f4c40cbcc0320f70781066862ccf88387fd54c7f51d399d7376d847ed49e1"}
Jul 08 16:42:12.600118 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:12.600085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f8gsq" event={"ID":"52c54020-3aeb-47f0-9395-faebb4a13865","Type":"ContainerStarted","Data":"577bf528ebc96fbb94218f348ee20b579dc22988087882a766dba7cbb2ea7ded"}
Jul 08 16:42:12.622721 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:42:12.622665    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-f8gsq" podStartSLOduration=2.904015974 podStartE2EDuration="13.622651194s" totalImagesPullingTime="8.969327922s" totalInitContainerRuntime="1.749307298s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:12.621809082 +0000 UTC m=+502.511224091" watchObservedRunningTime="2026-07-08 16:42:12.622651194 +0000 UTC m=+502.512066204"
Jul 08 16:43:17.833834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:17.833800    2559 generic.go:356] "Generic (PLEG): container finished" podID="52c54020-3aeb-47f0-9395-faebb4a13865" containerID="577bf528ebc96fbb94218f348ee20b579dc22988087882a766dba7cbb2ea7ded" exitCode=0
Jul 08 16:43:17.834229 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:17.833862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f8gsq" event={"ID":"52c54020-3aeb-47f0-9395-faebb4a13865","Type":"ContainerDied","Data":"577bf528ebc96fbb94218f348ee20b579dc22988087882a766dba7cbb2ea7ded"}
Jul 08 16:43:18.960848 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:18.960824    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f8gsq"
Jul 08 16:43:19.029356 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.029331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c54020-3aeb-47f0-9395-faebb4a13865-model-cache\") pod \"52c54020-3aeb-47f0-9395-faebb4a13865\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") "
Jul 08 16:43:19.029465 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.029397    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sgqr\" (UniqueName: \"kubernetes.io/projected/52c54020-3aeb-47f0-9395-faebb4a13865-kube-api-access-9sgqr\") pod \"52c54020-3aeb-47f0-9395-faebb4a13865\" (UID: \"52c54020-3aeb-47f0-9395-faebb4a13865\") "
Jul 08 16:43:19.031402 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.031374    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c54020-3aeb-47f0-9395-faebb4a13865-kube-api-access-9sgqr" (OuterVolumeSpecName: "kube-api-access-9sgqr") pod "52c54020-3aeb-47f0-9395-faebb4a13865" (UID: "52c54020-3aeb-47f0-9395-faebb4a13865"). InnerVolumeSpecName "kube-api-access-9sgqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:43:19.047689 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.047637    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c54020-3aeb-47f0-9395-faebb4a13865-model-cache" (OuterVolumeSpecName: "model-cache") pod "52c54020-3aeb-47f0-9395-faebb4a13865" (UID: "52c54020-3aeb-47f0-9395-faebb4a13865"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:43:19.130544 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.130470    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c54020-3aeb-47f0-9395-faebb4a13865-model-cache\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:43:19.130544 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.130497    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sgqr\" (UniqueName: \"kubernetes.io/projected/52c54020-3aeb-47f0-9395-faebb4a13865-kube-api-access-9sgqr\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:43:19.843516 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.843484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f8gsq" event={"ID":"52c54020-3aeb-47f0-9395-faebb4a13865","Type":"ContainerDied","Data":"ecbc2a27fa8599b99ef4a56882facdccac92b3aa4ec43a3d6bb0c2fc5dc8c806"}
Jul 08 16:43:19.843516 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.843523    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecbc2a27fa8599b99ef4a56882facdccac92b3aa4ec43a3d6bb0c2fc5dc8c806"
Jul 08 16:43:19.843750 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:19.843496    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f8gsq"
Jul 08 16:43:20.791934 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:20.791906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:50.568322 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:50.568294    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:43:50.568877 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:50.568789    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:43:53.910397 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.910365    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-pfttt"]
Jul 08 16:43:53.910875 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.910635    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c54020-3aeb-47f0-9395-faebb4a13865" containerName="s3-init"
Jul 08 16:43:53.910875 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.910645    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c54020-3aeb-47f0-9395-faebb4a13865" containerName="s3-init"
Jul 08 16:43:53.910875 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.910665    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c54020-3aeb-47f0-9395-faebb4a13865" containerName="download-hf-model"
Jul 08 16:43:53.910875 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.910672    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c54020-3aeb-47f0-9395-faebb4a13865" containerName="download-hf-model"
Jul 08 16:43:53.910875 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.910737    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c54020-3aeb-47f0-9395-faebb4a13865" containerName="s3-init"
Jul 08 16:43:53.913763 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.913747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:53.915666 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.915644    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:43:53.915782 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.915708    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 08 16:43:53.916091 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.916070    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:43:53.916183 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.916070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cjbv7\""
Jul 08 16:43:53.921495 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.921471    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-pfttt"]
Jul 08 16:43:53.977885 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.977861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-model-cache\") pod \"s3-tls-init-custom-pfttt\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") " pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:53.978004 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:53.977894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knhf\" (UniqueName: \"kubernetes.io/projected/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-kube-api-access-6knhf\") pod \"s3-tls-init-custom-pfttt\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") " pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:54.078839 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.078814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-model-cache\") pod \"s3-tls-init-custom-pfttt\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") " pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:54.078947 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.078848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knhf\" (UniqueName: \"kubernetes.io/projected/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-kube-api-access-6knhf\") pod \"s3-tls-init-custom-pfttt\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") " pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:54.079199 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.079181    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-model-cache\") pod \"s3-tls-init-custom-pfttt\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") " pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:54.087863 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.087844    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knhf\" (UniqueName: \"kubernetes.io/projected/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-kube-api-access-6knhf\") pod \"s3-tls-init-custom-pfttt\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") " pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:54.223792 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.223736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:43:54.346023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.345994    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-pfttt"]
Jul 08 16:43:54.349428 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:43:54.349398    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15f08b71_1d2c_4f82_a9c9_cb6058a4b250.slice/crio-f981f2799bf4c646979fd01c7fdef4dad073727c7e579dfefc64098500575e2b WatchSource:0}: Error finding container f981f2799bf4c646979fd01c7fdef4dad073727c7e579dfefc64098500575e2b: Status 404 returned error can't find the container with id f981f2799bf4c646979fd01c7fdef4dad073727c7e579dfefc64098500575e2b
Jul 08 16:43:54.966118 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.966083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pfttt" event={"ID":"15f08b71-1d2c-4f82-a9c9-cb6058a4b250","Type":"ContainerStarted","Data":"d8073a302ffa0770a9104bdacb7cc22ad759c58453f66b17de6a11f12a4d8acf"}
Jul 08 16:43:54.966118 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:54.966124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pfttt" event={"ID":"15f08b71-1d2c-4f82-a9c9-cb6058a4b250","Type":"ContainerStarted","Data":"f981f2799bf4c646979fd01c7fdef4dad073727c7e579dfefc64098500575e2b"}
Jul 08 16:43:56.974959 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:56.974930    2559 generic.go:356] "Generic (PLEG): container finished" podID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerID="d8073a302ffa0770a9104bdacb7cc22ad759c58453f66b17de6a11f12a4d8acf" exitCode=0
Jul 08 16:43:56.975325 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:56.974959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pfttt" event={"ID":"15f08b71-1d2c-4f82-a9c9-cb6058a4b250","Type":"ContainerDied","Data":"d8073a302ffa0770a9104bdacb7cc22ad759c58453f66b17de6a11f12a4d8acf"}
Jul 08 16:43:57.982250 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:57.982222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pfttt" event={"ID":"15f08b71-1d2c-4f82-a9c9-cb6058a4b250","Type":"ContainerStarted","Data":"8d7ff417a78c36c6416f995965e790a6545152b76263f0d87272301c07006181"}
Jul 08 16:43:57.998441 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:43:57.998331    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-pfttt" podStartSLOduration=3.14352895 podStartE2EDuration="4.998317802s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.854788852s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:57.99797127 +0000 UTC m=+607.887386280" watchObservedRunningTime="2026-07-08 16:43:57.998317802 +0000 UTC m=+607.887732812"
Jul 08 16:44:04.013957 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:04.013924    2559 generic.go:356] "Generic (PLEG): container finished" podID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerID="8d7ff417a78c36c6416f995965e790a6545152b76263f0d87272301c07006181" exitCode=0
Jul 08 16:44:04.014320 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:04.013997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pfttt" event={"ID":"15f08b71-1d2c-4f82-a9c9-cb6058a4b250","Type":"ContainerDied","Data":"8d7ff417a78c36c6416f995965e790a6545152b76263f0d87272301c07006181"}
Jul 08 16:44:04.835495 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:04.835465    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:05.169640 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.169617    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:44:05.265173 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.265150    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-model-cache\") pod \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") "
Jul 08 16:44:05.265278 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.265221    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6knhf\" (UniqueName: \"kubernetes.io/projected/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-kube-api-access-6knhf\") pod \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\" (UID: \"15f08b71-1d2c-4f82-a9c9-cb6058a4b250\") "
Jul 08 16:44:05.267142 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.267120    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-kube-api-access-6knhf" (OuterVolumeSpecName: "kube-api-access-6knhf") pod "15f08b71-1d2c-4f82-a9c9-cb6058a4b250" (UID: "15f08b71-1d2c-4f82-a9c9-cb6058a4b250"). InnerVolumeSpecName "kube-api-access-6knhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:44:05.290104 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.290082    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-model-cache" (OuterVolumeSpecName: "model-cache") pod "15f08b71-1d2c-4f82-a9c9-cb6058a4b250" (UID: "15f08b71-1d2c-4f82-a9c9-cb6058a4b250"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:44:05.366634 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.366610    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knhf\" (UniqueName: \"kubernetes.io/projected/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-kube-api-access-6knhf\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:44:05.366634 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:05.366632    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15f08b71-1d2c-4f82-a9c9-cb6058a4b250-model-cache\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:44:06.022802 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:06.022774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-pfttt" event={"ID":"15f08b71-1d2c-4f82-a9c9-cb6058a4b250","Type":"ContainerDied","Data":"f981f2799bf4c646979fd01c7fdef4dad073727c7e579dfefc64098500575e2b"}
Jul 08 16:44:06.022802 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:06.022801    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-pfttt"
Jul 08 16:44:06.022987 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:06.022807    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f981f2799bf4c646979fd01c7fdef4dad073727c7e579dfefc64098500575e2b"
Jul 08 16:44:06.637575 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:06.637545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:07.296588 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:07.296555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:09.187448 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.187421    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-hlzbj"]
Jul 08 16:44:09.187834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.187721    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerName="s3-tls-init-custom"
Jul 08 16:44:09.187834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.187734    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerName="s3-tls-init-custom"
Jul 08 16:44:09.187834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.187753    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerName="download-hf-model"
Jul 08 16:44:09.187834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.187758    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerName="download-hf-model"
Jul 08 16:44:09.187834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.187807    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f08b71-1d2c-4f82-a9c9-cb6058a4b250" containerName="s3-tls-init-custom"
Jul 08 16:44:09.191600 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.191584    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.193340 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.193314    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 08 16:44:09.193470 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.193357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cjbv7\""
Jul 08 16:44:09.193470 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.193371    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:44:09.193750 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.193735    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:44:09.197730 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.197706    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hlzbj"]
Jul 08 16:44:09.292896 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.292867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5qb\" (UniqueName: \"kubernetes.io/projected/bd22161f-4aaa-40f7-b4db-6268b613a2af-kube-api-access-9t5qb\") pod \"s3-tls-init-serving-hlzbj\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") " pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.293045 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.292918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd22161f-4aaa-40f7-b4db-6268b613a2af-model-cache\") pod \"s3-tls-init-serving-hlzbj\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") " pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.394206 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.394169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5qb\" (UniqueName: \"kubernetes.io/projected/bd22161f-4aaa-40f7-b4db-6268b613a2af-kube-api-access-9t5qb\") pod \"s3-tls-init-serving-hlzbj\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") " pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.394364 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.394229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd22161f-4aaa-40f7-b4db-6268b613a2af-model-cache\") pod \"s3-tls-init-serving-hlzbj\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") " pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.394551 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.394534    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd22161f-4aaa-40f7-b4db-6268b613a2af-model-cache\") pod \"s3-tls-init-serving-hlzbj\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") " pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.403081 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.403055    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5qb\" (UniqueName: \"kubernetes.io/projected/bd22161f-4aaa-40f7-b4db-6268b613a2af-kube-api-access-9t5qb\") pod \"s3-tls-init-serving-hlzbj\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") " pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.500635 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.500611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:09.627353 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:09.627303    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hlzbj"]
Jul 08 16:44:09.629747 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:44:09.629722    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd22161f_4aaa_40f7_b4db_6268b613a2af.slice/crio-c46e616aa5b0d2cc9895f87a2c18b457d3b292f79b41ae429fe5963b2191071f WatchSource:0}: Error finding container c46e616aa5b0d2cc9895f87a2c18b457d3b292f79b41ae429fe5963b2191071f: Status 404 returned error can't find the container with id c46e616aa5b0d2cc9895f87a2c18b457d3b292f79b41ae429fe5963b2191071f
Jul 08 16:44:10.038021 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:10.037989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hlzbj" event={"ID":"bd22161f-4aaa-40f7-b4db-6268b613a2af","Type":"ContainerStarted","Data":"96c1c92eb93085e6bceb9b3691761f12778ccf4bb6ae9147f1376c4343a858ed"}
Jul 08 16:44:10.038021 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:10.038027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hlzbj" event={"ID":"bd22161f-4aaa-40f7-b4db-6268b613a2af","Type":"ContainerStarted","Data":"c46e616aa5b0d2cc9895f87a2c18b457d3b292f79b41ae429fe5963b2191071f"}
Jul 08 16:44:12.046943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:12.046914    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerID="96c1c92eb93085e6bceb9b3691761f12778ccf4bb6ae9147f1376c4343a858ed" exitCode=0
Jul 08 16:44:12.047289 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:12.046953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hlzbj" event={"ID":"bd22161f-4aaa-40f7-b4db-6268b613a2af","Type":"ContainerDied","Data":"96c1c92eb93085e6bceb9b3691761f12778ccf4bb6ae9147f1376c4343a858ed"}
Jul 08 16:44:13.052511 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:13.052445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hlzbj" event={"ID":"bd22161f-4aaa-40f7-b4db-6268b613a2af","Type":"ContainerStarted","Data":"cb4080fe9900e14d5aa387fc96d4bc1300b8d52c738c08ff5c730201fe0fff3e"}
Jul 08 16:44:13.072347 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:13.072306    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-hlzbj" podStartSLOduration=1.946651514 podStartE2EDuration="4.07229218s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.125640666s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:13.070813409 +0000 UTC m=+622.960228418" watchObservedRunningTime="2026-07-08 16:44:13.07229218 +0000 UTC m=+622.961707189"
Jul 08 16:44:18.073971 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:18.073938    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerID="cb4080fe9900e14d5aa387fc96d4bc1300b8d52c738c08ff5c730201fe0fff3e" exitCode=0
Jul 08 16:44:18.073971 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:18.073973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hlzbj" event={"ID":"bd22161f-4aaa-40f7-b4db-6268b613a2af","Type":"ContainerDied","Data":"cb4080fe9900e14d5aa387fc96d4bc1300b8d52c738c08ff5c730201fe0fff3e"}
Jul 08 16:44:19.225217 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.225189    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:19.274031 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.274007    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd22161f-4aaa-40f7-b4db-6268b613a2af-model-cache\") pod \"bd22161f-4aaa-40f7-b4db-6268b613a2af\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") "
Jul 08 16:44:19.274165 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.274038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t5qb\" (UniqueName: \"kubernetes.io/projected/bd22161f-4aaa-40f7-b4db-6268b613a2af-kube-api-access-9t5qb\") pod \"bd22161f-4aaa-40f7-b4db-6268b613a2af\" (UID: \"bd22161f-4aaa-40f7-b4db-6268b613a2af\") "
Jul 08 16:44:19.276064 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.276034    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd22161f-4aaa-40f7-b4db-6268b613a2af-kube-api-access-9t5qb" (OuterVolumeSpecName: "kube-api-access-9t5qb") pod "bd22161f-4aaa-40f7-b4db-6268b613a2af" (UID: "bd22161f-4aaa-40f7-b4db-6268b613a2af"). InnerVolumeSpecName "kube-api-access-9t5qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:44:19.296301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.296275    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd22161f-4aaa-40f7-b4db-6268b613a2af-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd22161f-4aaa-40f7-b4db-6268b613a2af" (UID: "bd22161f-4aaa-40f7-b4db-6268b613a2af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:44:19.374672 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.374618    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd22161f-4aaa-40f7-b4db-6268b613a2af-model-cache\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:44:19.374672 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:19.374637    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5qb\" (UniqueName: \"kubernetes.io/projected/bd22161f-4aaa-40f7-b4db-6268b613a2af-kube-api-access-9t5qb\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:44:20.082470 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:20.082445    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hlzbj"
Jul 08 16:44:20.082639 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:20.082470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hlzbj" event={"ID":"bd22161f-4aaa-40f7-b4db-6268b613a2af","Type":"ContainerDied","Data":"c46e616aa5b0d2cc9895f87a2c18b457d3b292f79b41ae429fe5963b2191071f"}
Jul 08 16:44:20.082639 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:20.082499    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46e616aa5b0d2cc9895f87a2c18b457d3b292f79b41ae429fe5963b2191071f"
Jul 08 16:44:20.479140 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:20.479098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:20.579555 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:20.579527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:23.105333 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:23.105295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:25.633294 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:25.633265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:25.924016 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:25.923938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:26.033486 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:26.033459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:33.977096 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:33.977062    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:34.112424 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.112387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:34.117095 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.117070    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"]
Jul 08 16:44:34.117390 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.117374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerName="download-hf-model"
Jul 08 16:44:34.117462 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.117393    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerName="download-hf-model"
Jul 08 16:44:34.117462 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.117436    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerName="s3-tls-init-serving"
Jul 08 16:44:34.117462 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.117446    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerName="s3-tls-init-serving"
Jul 08 16:44:34.117613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.117508    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd22161f-4aaa-40f7-b4db-6268b613a2af" containerName="s3-tls-init-serving"
Jul 08 16:44:34.122258 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.122240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.131531 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.131507    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 08 16:44:34.131531 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.131521    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 08 16:44:34.131649 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.131506    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:44:34.131649 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.131521    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-924bj\""
Jul 08 16:44:34.131649 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.131506    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:44:34.132762 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.132747    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:44:34.132837 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.132822    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:44:34.179772 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.179749    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.179881 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.179791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9907872f-7973-4a5d-b6d3-7607955b5383-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.179881 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.179862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9907872f-7973-4a5d-b6d3-7607955b5383-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.179999 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.179905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgxf\" (UniqueName: \"kubernetes.io/projected/9907872f-7973-4a5d-b6d3-7607955b5383-kube-api-access-jfgxf\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.179999 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.179939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.233968 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.233945    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"]
Jul 08 16:44:34.280671 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.280648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9907872f-7973-4a5d-b6d3-7607955b5383-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.280771 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.280694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgxf\" (UniqueName: \"kubernetes.io/projected/9907872f-7973-4a5d-b6d3-7607955b5383-kube-api-access-jfgxf\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.280771 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.280717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.280771 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.280741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.280922 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.280775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9907872f-7973-4a5d-b6d3-7607955b5383-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.281179 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.281162    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9907872f-7973-4a5d-b6d3-7607955b5383-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.281462 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.281431    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.281566 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.281507    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.283318 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.283298    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9907872f-7973-4a5d-b6d3-7607955b5383-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.298920 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.298898    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgxf\" (UniqueName: \"kubernetes.io/projected/9907872f-7973-4a5d-b6d3-7607955b5383-kube-api-access-jfgxf\") pod \"isvc-sklearn-batcher-predictor-679bd44578-pbxtw\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.432474 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.432446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:34.574031 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:34.574002    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"]
Jul 08 16:44:34.577149 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:44:34.577123    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9907872f_7973_4a5d_b6d3_7607955b5383.slice/crio-b41030ef5c000c7be9d187dd56cbbd73f7b87057b507018f15ba2a812404d6d3 WatchSource:0}: Error finding container b41030ef5c000c7be9d187dd56cbbd73f7b87057b507018f15ba2a812404d6d3: Status 404 returned error can't find the container with id b41030ef5c000c7be9d187dd56cbbd73f7b87057b507018f15ba2a812404d6d3
Jul 08 16:44:35.147090 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:35.147052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerStarted","Data":"165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d"}
Jul 08 16:44:35.147441 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:35.147133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerStarted","Data":"b41030ef5c000c7be9d187dd56cbbd73f7b87057b507018f15ba2a812404d6d3"}
Jul 08 16:44:39.163291 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:39.163260    2559 generic.go:356] "Generic (PLEG): container finished" podID="9907872f-7973-4a5d-b6d3-7607955b5383" containerID="165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d" exitCode=0
Jul 08 16:44:39.163670 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:39.163313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerDied","Data":"165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d"}
Jul 08 16:44:53.235167 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:53.235132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerStarted","Data":"b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b"}
Jul 08 16:44:55.245743 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:55.245704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerStarted","Data":"8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d"}
Jul 08 16:44:58.261531 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:58.261498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerStarted","Data":"6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910"}
Jul 08 16:44:58.262014 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:58.261717    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:58.262014 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:58.261867    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:58.263238 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:58.263203    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:44:58.292147 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:58.292072    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podStartSLOduration=1.7152836599999999 podStartE2EDuration="24.292058565s" totalImagesPullingTime="18.730728097s" totalInitContainerRuntime="3.846046808s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:58.291044621 +0000 UTC m=+668.180459658" watchObservedRunningTime="2026-07-08 16:44:58.292058565 +0000 UTC m=+668.181473575"
Jul 08 16:44:59.265176 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:59.265143    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:44:59.265626 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:59.265240    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:44:59.266180 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:59.266154    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:44:59.269019 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:44:59.268998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:45:00.269188 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:00.269156    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:00.269577 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:00.269446    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:01.272986 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:01.272950    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:01.273447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:01.273293    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:11.273582 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:11.273523    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:11.274089 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:11.273920    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:21.273153 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:21.273110    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:21.273647 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:21.273625    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:31.273655 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:31.273606    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:31.274163 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:31.274097    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:41.273789 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:41.273740    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:41.274279 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:41.274255    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:51.273196 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:51.273104    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:45:51.273640 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:51.273477    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:45:53.453832 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:53.453802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:45:55.414153 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:45:55.414118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:01.273133 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:01.273094    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:46:01.273599 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:01.273575    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:03.669560 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:03.669531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:03.670014 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:03.669607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:19.192256 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.192217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:19.210229 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.210198    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"]
Jul 08 16:46:19.210499 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.210468    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" containerID="cri-o://6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910" gracePeriod=30
Jul 08 16:46:19.210643 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.210514    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" containerID="cri-o://8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d" gracePeriod=30
Jul 08 16:46:19.210643 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.210465    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" containerID="cri-o://b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b" gracePeriod=30
Jul 08 16:46:19.265945 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.265899    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:19.278520 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.278478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:19.279550 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.279533    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"]
Jul 08 16:46:19.283211 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.283195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.285719 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.285696    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 08 16:46:19.285933 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.285916    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 08 16:46:19.298628 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.298583    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"]
Jul 08 16:46:19.381132 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.381105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.381234 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.381155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcps2\" (UniqueName: \"kubernetes.io/projected/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-api-access-tcps2\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.381234 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.381191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.381332 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.381312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.381374 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.381352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.482304 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.482278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.482443 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.482318    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcps2\" (UniqueName: \"kubernetes.io/projected/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-api-access-tcps2\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.482443 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.482342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.482443 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.482397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.482443 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.482415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.482734 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:46:19.482603    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 08 16:46:19.482734 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:46:19.482669    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls podName:8b92cda0-7ad8-44d0-8f33-eabcf52d8c90 nodeName:}" failed. No retries permitted until 2026-07-08 16:46:19.98265324 +0000 UTC m=+749.872068232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" (UID: "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 08 16:46:19.482851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.482826    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.483155 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.483132    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.483229 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.483136    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.492387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.492367    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcps2\" (UniqueName: \"kubernetes.io/projected/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-api-access-tcps2\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.577355 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.577324    2559 generic.go:356] "Generic (PLEG): container finished" podID="9907872f-7973-4a5d-b6d3-7607955b5383" containerID="8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d" exitCode=2
Jul 08 16:46:19.577467 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.577394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerDied","Data":"8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d"}
Jul 08 16:46:19.819820 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.819742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:19.985967 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.985940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:19.988229 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:19.988208    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:20.193603 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:20.193527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:20.322914 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:20.322889    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"]
Jul 08 16:46:20.324795 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:46:20.324772    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b92cda0_7ad8_44d0_8f33_eabcf52d8c90.slice/crio-89b852fde28aef00c1b8253f4e5b7664e2799d002c7604c69ff6032abd523a97 WatchSource:0}: Error finding container 89b852fde28aef00c1b8253f4e5b7664e2799d002c7604c69ff6032abd523a97: Status 404 returned error can't find the container with id 89b852fde28aef00c1b8253f4e5b7664e2799d002c7604c69ff6032abd523a97
Jul 08 16:46:20.583247 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:20.583211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerStarted","Data":"addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af"}
Jul 08 16:46:20.583406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:20.583294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerStarted","Data":"89b852fde28aef00c1b8253f4e5b7664e2799d002c7604c69ff6032abd523a97"}
Jul 08 16:46:23.597410 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:23.597335    2559 generic.go:356] "Generic (PLEG): container finished" podID="9907872f-7973-4a5d-b6d3-7607955b5383" containerID="b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b" exitCode=0
Jul 08 16:46:23.597762 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:23.597411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerDied","Data":"b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b"}
Jul 08 16:46:23.670008 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:23.669974    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:46:23.670313 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:23.670294    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:24.265699 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:24.265649    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:24.603139 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:24.603111    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerID="addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af" exitCode=0
Jul 08 16:46:24.603594 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:24.603190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerDied","Data":"addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af"}
Jul 08 16:46:25.615559 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.615524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerStarted","Data":"140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda"}
Jul 08 16:46:25.615960 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.615619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerStarted","Data":"9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa"}
Jul 08 16:46:25.615960 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.615647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerStarted","Data":"890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19"}
Jul 08 16:46:25.616055 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.615968    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:25.616142 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.616124    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:25.617173 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.617148    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:46:25.641410 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:25.641369    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podStartSLOduration=2.969760046 podStartE2EDuration="6.641358267s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.671598221s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:25.638902179 +0000 UTC m=+755.528317189" watchObservedRunningTime="2026-07-08 16:46:25.641358267 +0000 UTC m=+755.530773277"
Jul 08 16:46:26.620050 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:26.620012    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:26.620441 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:26.620055    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:46:26.621241 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:26.621215    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:27.624036 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:27.623992    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:46:27.624441 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:27.624371    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:29.265815 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:29.265781    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:29.266211 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:29.265895    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:32.628728 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:32.628698    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:46:32.629292 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:32.629262    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:46:32.629517 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:32.629489    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:33.669558 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:33.669509    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:46:33.669981 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:33.669883    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:34.265767 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:34.265725    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:39.265837 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:39.265792    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:42.629727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:42.629668    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:46:42.630145 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:42.630108    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:43.669727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:43.669658    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 08 16:46:43.670155 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:43.669835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:43.670155 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:43.669922    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:46:43.670155 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:43.670027    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:44.266177 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:44.266136    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:49.266217 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.266182    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 08 16:46:49.353095 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.353073    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:49.511736 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.511705    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9907872f-7973-4a5d-b6d3-7607955b5383-kserve-provision-location\") pod \"9907872f-7973-4a5d-b6d3-7607955b5383\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") "
Jul 08 16:46:49.511892 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.511747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-cabundle-cert\") pod \"9907872f-7973-4a5d-b6d3-7607955b5383\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") "
Jul 08 16:46:49.511892 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.511804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-kube-rbac-proxy-sar-config\") pod \"9907872f-7973-4a5d-b6d3-7607955b5383\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") "
Jul 08 16:46:49.511892 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.511829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9907872f-7973-4a5d-b6d3-7607955b5383-proxy-tls\") pod \"9907872f-7973-4a5d-b6d3-7607955b5383\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") "
Jul 08 16:46:49.511892 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.511888    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfgxf\" (UniqueName: \"kubernetes.io/projected/9907872f-7973-4a5d-b6d3-7607955b5383-kube-api-access-jfgxf\") pod \"9907872f-7973-4a5d-b6d3-7607955b5383\" (UID: \"9907872f-7973-4a5d-b6d3-7607955b5383\") "
Jul 08 16:46:49.512124 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.512095    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9907872f-7973-4a5d-b6d3-7607955b5383-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9907872f-7973-4a5d-b6d3-7607955b5383" (UID: "9907872f-7973-4a5d-b6d3-7607955b5383"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:49.512178 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.512162    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9907872f-7973-4a5d-b6d3-7607955b5383" (UID: "9907872f-7973-4a5d-b6d3-7607955b5383"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:49.512217 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.512170    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9907872f-7973-4a5d-b6d3-7607955b5383" (UID: "9907872f-7973-4a5d-b6d3-7607955b5383"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:49.514226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.514201    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9907872f-7973-4a5d-b6d3-7607955b5383-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9907872f-7973-4a5d-b6d3-7607955b5383" (UID: "9907872f-7973-4a5d-b6d3-7607955b5383"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:46:49.514226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.514215    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9907872f-7973-4a5d-b6d3-7607955b5383-kube-api-access-jfgxf" (OuterVolumeSpecName: "kube-api-access-jfgxf") pod "9907872f-7973-4a5d-b6d3-7607955b5383" (UID: "9907872f-7973-4a5d-b6d3-7607955b5383"). InnerVolumeSpecName "kube-api-access-jfgxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:49.613406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.613368    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfgxf\" (UniqueName: \"kubernetes.io/projected/9907872f-7973-4a5d-b6d3-7607955b5383-kube-api-access-jfgxf\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:46:49.613406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.613399    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9907872f-7973-4a5d-b6d3-7607955b5383-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:46:49.613406 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.613408    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:46:49.613623 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.613418    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9907872f-7973-4a5d-b6d3-7607955b5383-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:46:49.613623 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.613428    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9907872f-7973-4a5d-b6d3-7607955b5383-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:46:49.709418 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.709381    2559 generic.go:356] "Generic (PLEG): container finished" podID="9907872f-7973-4a5d-b6d3-7607955b5383" containerID="6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910" exitCode=0
Jul 08 16:46:49.709552 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.709458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerDied","Data":"6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910"}
Jul 08 16:46:49.709552 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.709520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw" event={"ID":"9907872f-7973-4a5d-b6d3-7607955b5383","Type":"ContainerDied","Data":"b41030ef5c000c7be9d187dd56cbbd73f7b87057b507018f15ba2a812404d6d3"}
Jul 08 16:46:49.709552 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.709466    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"
Jul 08 16:46:49.709552 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.709551    2559 scope.go:117] "RemoveContainer" containerID="6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910"
Jul 08 16:46:49.718255 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.718236    2559 scope.go:117] "RemoveContainer" containerID="8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d"
Jul 08 16:46:49.725377 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.725360    2559 scope.go:117] "RemoveContainer" containerID="b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b"
Jul 08 16:46:49.732635 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.732614    2559 scope.go:117] "RemoveContainer" containerID="165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d"
Jul 08 16:46:49.733064 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.733043    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"]
Jul 08 16:46:49.738898 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.738874    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-679bd44578-pbxtw"]
Jul 08 16:46:49.740423 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.740409    2559 scope.go:117] "RemoveContainer" containerID="6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910"
Jul 08 16:46:49.740690 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:46:49.740662    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910\": container with ID starting with 6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910 not found: ID does not exist" containerID="6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910"
Jul 08 16:46:49.740773 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.740707    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910"} err="failed to get container status \"6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910\": rpc error: code = NotFound desc = could not find container \"6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910\": container with ID starting with 6c41a88b0b0b2b20c23bdb9052f3d2dfe2703a77fb22250f82e8ec4918e41910 not found: ID does not exist"
Jul 08 16:46:49.740773 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.740732    2559 scope.go:117] "RemoveContainer" containerID="8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d"
Jul 08 16:46:49.740990 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:46:49.740975    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d\": container with ID starting with 8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d not found: ID does not exist" containerID="8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d"
Jul 08 16:46:49.741045 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.740997    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d"} err="failed to get container status \"8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d\": rpc error: code = NotFound desc = could not find container \"8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d\": container with ID starting with 8dcd11c380d6b2f73459a27d55ea1a4a11fb04d35eceab634842ddc77bda539d not found: ID does not exist"
Jul 08 16:46:49.741045 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.741018    2559 scope.go:117] "RemoveContainer" containerID="b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b"
Jul 08 16:46:49.741250 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:46:49.741234    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b\": container with ID starting with b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b not found: ID does not exist" containerID="b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b"
Jul 08 16:46:49.741315 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.741256    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b"} err="failed to get container status \"b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b\": rpc error: code = NotFound desc = could not find container \"b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b\": container with ID starting with b78daf7f72f637a092b32769d7d905ce4cf73256abcbad2dbe5f8450c80f227b not found: ID does not exist"
Jul 08 16:46:49.741315 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.741274    2559 scope.go:117] "RemoveContainer" containerID="165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d"
Jul 08 16:46:49.741481 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:46:49.741465    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d\": container with ID starting with 165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d not found: ID does not exist" containerID="165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d"
Jul 08 16:46:49.741527 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:49.741487    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d"} err="failed to get container status \"165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d\": rpc error: code = NotFound desc = could not find container \"165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d\": container with ID starting with 165d35e08a158572407f545a25879b5b2743450c121b094c5e53897fcec8e41d not found: ID does not exist"
Jul 08 16:46:50.673800 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:50.673766    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" path="/var/lib/kubelet/pods/9907872f-7973-4a5d-b6d3-7607955b5383/volumes"
Jul 08 16:46:52.629339 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:52.629295    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:46:52.629771 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:46:52.629729    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:47:02.629330 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:02.629284    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:47:02.629819 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:02.629701    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:47:08.943565 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:08.943523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:09.572378 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:09.572348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:12.630022 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:12.629977    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:47:12.630439 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:12.630417    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:47:22.629489 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:22.629449    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:47:22.629878 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:22.629820    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:47:32.630120 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:32.630081    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:47:32.630570 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:32.630548    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:47:42.629818 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:42.629790    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:47:42.630290 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:42.629921    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:47:54.347284 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.347251    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"]
Jul 08 16:47:54.347776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.347517    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" containerID="cri-o://890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19" gracePeriod=30
Jul 08 16:47:54.347776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.347560    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" containerID="cri-o://9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa" gracePeriod=30
Jul 08 16:47:54.347776 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.347594    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" containerID="cri-o://140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda" gracePeriod=30
Jul 08 16:47:54.364361 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.364326    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:54.440382 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440355    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"]
Jul 08 16:47:54.440657 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440645    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440660    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440672    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440693    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440713    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440720    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440726    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="storage-initializer"
Jul 08 16:47:54.440731 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440731    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="storage-initializer"
Jul 08 16:47:54.440998 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440805    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="agent"
Jul 08 16:47:54.440998 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440813    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kube-rbac-proxy"
Jul 08 16:47:54.440998 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.440819    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9907872f-7973-4a5d-b6d3-7607955b5383" containerName="kserve-container"
Jul 08 16:47:54.443801 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.443786    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.445904 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.445883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 08 16:47:54.445998 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.445920    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 08 16:47:54.455667 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.455645    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"]
Jul 08 16:47:54.468293 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.468269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:54.605049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.604977    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.605049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.605017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.605049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.605042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtzk\" (UniqueName: \"kubernetes.io/projected/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-api-access-jhtzk\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.705915 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.705892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.706044 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.705920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtzk\" (UniqueName: \"kubernetes.io/projected/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-api-access-jhtzk\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.706044 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.705999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.706150 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:47:54.706112    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 08 16:47:54.706187 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:47:54.706178    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls podName:b642155d-530a-4125-ab41-5d25bf7b6d30 nodeName:}" failed. No retries permitted until 2026-07-08 16:47:55.206157273 +0000 UTC m=+845.095572278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-mcxtb" (UID: "b642155d-530a-4125-ab41-5d25bf7b6d30") : secret "message-dumper-predictor-serving-cert" not found
Jul 08 16:47:54.706615 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.706597    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.716403 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.716378    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtzk\" (UniqueName: \"kubernetes.io/projected/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-api-access-jhtzk\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:54.961594 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.961524    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerID="9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa" exitCode=2
Jul 08 16:47:54.961739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:54.961596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerDied","Data":"9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa"}
Jul 08 16:47:55.024738 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.024706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:55.209138 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.209109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:55.211380 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.211352    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mcxtb\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:55.353280 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.353257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:55.478839 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.478809    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"]
Jul 08 16:47:55.481024 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:47:55.480993    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb642155d_530a_4125_ab41_5d25bf7b6d30.slice/crio-c7f546af3c23b1fb1167d3810577c2d619bfb446ee7f950befced0e72bd16c2e WatchSource:0}: Error finding container c7f546af3c23b1fb1167d3810577c2d619bfb446ee7f950befced0e72bd16c2e: Status 404 returned error can't find the container with id c7f546af3c23b1fb1167d3810577c2d619bfb446ee7f950befced0e72bd16c2e
Jul 08 16:47:55.482961 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.482946    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:47:55.966842 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:55.966813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" event={"ID":"b642155d-530a-4125-ab41-5d25bf7b6d30","Type":"ContainerStarted","Data":"c7f546af3c23b1fb1167d3810577c2d619bfb446ee7f950befced0e72bd16c2e"}
Jul 08 16:47:56.973729 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:56.973628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" event={"ID":"b642155d-530a-4125-ab41-5d25bf7b6d30","Type":"ContainerStarted","Data":"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"}
Jul 08 16:47:56.974121 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:56.973741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" event={"ID":"b642155d-530a-4125-ab41-5d25bf7b6d30","Type":"ContainerStarted","Data":"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"}
Jul 08 16:47:56.974121 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:56.973769    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:56.995583 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:56.995541    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" podStartSLOduration=1.79819179 podStartE2EDuration="2.995527076s" totalImagesPullingTime="1.197335286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:47:56.995488616 +0000 UTC m=+846.884903626" watchObservedRunningTime="2026-07-08 16:47:56.995527076 +0000 UTC m=+846.884942080"
Jul 08 16:47:57.624265 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:57.624224    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 08 16:47:57.978047 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:57.977970    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:57.979621 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:57.979599    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:47:58.984839 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:58.984807    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerID="890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19" exitCode=0
Jul 08 16:47:58.985180 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:47:58.984878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerDied","Data":"890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19"}
Jul 08 16:48:02.624299 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:02.624259    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 08 16:48:02.629701 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:02.629645    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:48:02.629960 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:02.629935    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:04.993038 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:04.993008    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:48:07.624941 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:07.624902    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 08 16:48:07.625345 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:07.625023    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:48:12.624328 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:12.624277    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 08 16:48:12.629648 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:12.629618    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:48:12.630001 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:12.629977    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:14.450974 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.450922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:14.495373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.495343    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"]
Jul 08 16:48:14.498927 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.498911    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.500871 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.500849    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 08 16:48:14.500990 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.500849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 08 16:48:14.511090 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.511061    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"]
Jul 08 16:48:14.551041 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.551014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-cabundle-cert\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.551165 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.551050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpr68\" (UniqueName: \"kubernetes.io/projected/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-api-access-cpr68\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.551165 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.551074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.551165 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.551141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.551309 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.551188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kserve-provision-location\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.651729 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.651697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-cabundle-cert\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.651886 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.651736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr68\" (UniqueName: \"kubernetes.io/projected/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-api-access-cpr68\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.651886 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.651764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.652008 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:48:14.651904    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 08 16:48:14.652008 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:48:14.651949    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls podName:2af7ed25-87ce-42c2-928b-e69f0069b0c5 nodeName:}" failed. No retries permitted until 2026-07-08 16:48:15.151934912 +0000 UTC m=+865.041349900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls") pod "isvc-logger-predictor-5d78f49685-sx85z" (UID: "2af7ed25-87ce-42c2-928b-e69f0069b0c5") : secret "isvc-logger-predictor-serving-cert" not found
Jul 08 16:48:14.652129 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.651996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.652129 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.652039    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kserve-provision-location\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.652442 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.652418    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kserve-provision-location\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.652521 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.652499    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-cabundle-cert\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.652622 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.652606    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:14.662508 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:14.662489    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpr68\" (UniqueName: \"kubernetes.io/projected/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-api-access-cpr68\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:15.132301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:15.132264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:15.155659 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:15.155632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:15.157988 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:15.157963    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls\") pod \"isvc-logger-predictor-5d78f49685-sx85z\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:15.411142 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:15.411080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:15.546200 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:15.546170    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"]
Jul 08 16:48:15.548531 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:48:15.548503    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af7ed25_87ce_42c2_928b_e69f0069b0c5.slice/crio-76c6adcc63e94ce6801eb56a53a795c778483dd0bddab14569351f04920b7323 WatchSource:0}: Error finding container 76c6adcc63e94ce6801eb56a53a795c778483dd0bddab14569351f04920b7323: Status 404 returned error can't find the container with id 76c6adcc63e94ce6801eb56a53a795c778483dd0bddab14569351f04920b7323
Jul 08 16:48:16.051373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:16.051337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerStarted","Data":"c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194"}
Jul 08 16:48:16.051537 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:16.051412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerStarted","Data":"76c6adcc63e94ce6801eb56a53a795c778483dd0bddab14569351f04920b7323"}
Jul 08 16:48:17.624987 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:17.624952    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 08 16:48:20.069618 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:20.069591    2559 generic.go:356] "Generic (PLEG): container finished" podID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerID="c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194" exitCode=0
Jul 08 16:48:20.070006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:20.069671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerDied","Data":"c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194"}
Jul 08 16:48:21.075948 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.075910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerStarted","Data":"2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3"}
Jul 08 16:48:21.076288 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.075981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerStarted","Data":"40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5"}
Jul 08 16:48:21.076288 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.076006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerStarted","Data":"55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615"}
Jul 08 16:48:21.076384 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.076327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:21.076477 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.076460    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:21.077690 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.077649    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:21.100565 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:21.100523    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podStartSLOduration=3.474285439 podStartE2EDuration="7.100511279s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.62622584s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:21.098246625 +0000 UTC m=+870.987661632" watchObservedRunningTime="2026-07-08 16:48:21.100511279 +0000 UTC m=+870.989926289"
Jul 08 16:48:22.080366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.080334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:22.080803 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.080499    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:22.081482 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.081451    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:22.624347 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.624307    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 08 16:48:22.629173 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.629146    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:5000: connect: connection refused"
Jul 08 16:48:22.629271 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.629255    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:48:22.629452 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.629432    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:22.629523 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:22.629513    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:48:23.083809 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:23.083762    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:23.084248 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:23.084223    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:24.493478 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.493455    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:48:24.528521 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528493    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kserve-provision-location\") pod \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") "
Jul 08 16:48:24.528634 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528561    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcps2\" (UniqueName: \"kubernetes.io/projected/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-api-access-tcps2\") pod \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") "
Jul 08 16:48:24.528634 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-cabundle-cert\") pod \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") "
Jul 08 16:48:24.528752 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528665    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-rbac-proxy-sar-config\") pod \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") "
Jul 08 16:48:24.528752 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528721    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls\") pod \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\" (UID: \"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90\") "
Jul 08 16:48:24.528870 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528779    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" (UID: "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:24.528989 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.528966    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:48:24.529111 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.529038    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" (UID: "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:48:24.529111 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.529045    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" (UID: "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:48:24.530661 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.530633    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-api-access-tcps2" (OuterVolumeSpecName: "kube-api-access-tcps2") pod "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" (UID: "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90"). InnerVolumeSpecName "kube-api-access-tcps2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:48:24.530661 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.530643    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" (UID: "8b92cda0-7ad8-44d0-8f33-eabcf52d8c90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:48:24.629292 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.629232    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcps2\" (UniqueName: \"kubernetes.io/projected/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-api-access-tcps2\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:48:24.629292 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.629253    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:48:24.629292 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.629264    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:48:24.629292 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:24.629272    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:48:25.095520 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.095485    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerID="140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda" exitCode=0
Jul 08 16:48:25.095739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.095546    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"
Jul 08 16:48:25.095739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.095565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerDied","Data":"140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda"}
Jul 08 16:48:25.095739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.095643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls" event={"ID":"8b92cda0-7ad8-44d0-8f33-eabcf52d8c90","Type":"ContainerDied","Data":"89b852fde28aef00c1b8253f4e5b7664e2799d002c7604c69ff6032abd523a97"}
Jul 08 16:48:25.095739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.095703    2559 scope.go:117] "RemoveContainer" containerID="140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda"
Jul 08 16:48:25.103781 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.103763    2559 scope.go:117] "RemoveContainer" containerID="9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa"
Jul 08 16:48:25.110735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.110718    2559 scope.go:117] "RemoveContainer" containerID="890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19"
Jul 08 16:48:25.117733 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.117710    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"]
Jul 08 16:48:25.118206 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.118186    2559 scope.go:117] "RemoveContainer" containerID="addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af"
Jul 08 16:48:25.122420 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.122404    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-dbc474dd9-hx9ls"]
Jul 08 16:48:25.126089 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126072    2559 scope.go:117] "RemoveContainer" containerID="140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda"
Jul 08 16:48:25.126330 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:48:25.126314    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda\": container with ID starting with 140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda not found: ID does not exist" containerID="140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda"
Jul 08 16:48:25.126394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126337    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda"} err="failed to get container status \"140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda\": rpc error: code = NotFound desc = could not find container \"140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda\": container with ID starting with 140553e280d39079bcfa2deddca26caf9af2022044aaba14baa6e7197de64eda not found: ID does not exist"
Jul 08 16:48:25.126394 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126352    2559 scope.go:117] "RemoveContainer" containerID="9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa"
Jul 08 16:48:25.126568 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:48:25.126551    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa\": container with ID starting with 9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa not found: ID does not exist" containerID="9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa"
Jul 08 16:48:25.126614 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126573    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa"} err="failed to get container status \"9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa\": rpc error: code = NotFound desc = could not find container \"9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa\": container with ID starting with 9a452a0adc0ea041074c6264f95fb1bba1324222ad0adc3dccdae109ebea9afa not found: ID does not exist"
Jul 08 16:48:25.126614 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126589    2559 scope.go:117] "RemoveContainer" containerID="890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19"
Jul 08 16:48:25.126864 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:48:25.126845    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19\": container with ID starting with 890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19 not found: ID does not exist" containerID="890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19"
Jul 08 16:48:25.126918 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126868    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19"} err="failed to get container status \"890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19\": rpc error: code = NotFound desc = could not find container \"890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19\": container with ID starting with 890aa6db4c9c0c35b264b383992ab2219de5c6830b37cf3c50eed72ae33bfb19 not found: ID does not exist"
Jul 08 16:48:25.126918 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.126883    2559 scope.go:117] "RemoveContainer" containerID="addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af"
Jul 08 16:48:25.127158 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:48:25.127140    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af\": container with ID starting with addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af not found: ID does not exist" containerID="addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af"
Jul 08 16:48:25.127209 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:25.127161    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af"} err="failed to get container status \"addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af\": rpc error: code = NotFound desc = could not find container \"addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af\": container with ID starting with addb373b08cf676a6c0721b84a97a1fb13968fcd1903446255e5e5a8a9d0a8af not found: ID does not exist"
Jul 08 16:48:26.672705 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:26.672649    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" path="/var/lib/kubelet/pods/8b92cda0-7ad8-44d0-8f33-eabcf52d8c90/volumes"
Jul 08 16:48:28.088192 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:28.088160    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:48:28.088884 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:28.088837    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:28.089124 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:28.089095    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:38.088792 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:38.088743    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:38.089232 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:38.089206    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:48.088960 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:48.088866    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:48.089534 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:48.089494    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:48:50.593879 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:50.593850    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:48:50.594315 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:50.594054    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:48:58.088816 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:58.088663    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:48:58.091201 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:48:58.089168    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:08.089486 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:08.089445    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:49:08.089957 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:08.089838    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:18.089192 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:18.089151    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:49:18.089750 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:18.089724    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:28.088899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:28.088862    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:49:28.089350 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:28.089231    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:30.674228 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:30.674201    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:49:30.674658 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:30.674276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:49:39.470969 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.470934    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-mcxtb_b642155d-530a-4125-ab41-5d25bf7b6d30/kserve-container/0.log"
Jul 08 16:49:39.721084 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.720993    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"]
Jul 08 16:49:39.721350 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.721320    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" containerID="cri-o://55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615" gracePeriod=30
Jul 08 16:49:39.721453 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.721367    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" containerID="cri-o://2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3" gracePeriod=30
Jul 08 16:49:39.721512 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.721422    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" containerID="cri-o://40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5" gracePeriod=30
Jul 08 16:49:39.728287 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.728264    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:39.766352 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766321    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"]
Jul 08 16:49:39.766727 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766710    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="storage-initializer"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766729    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="storage-initializer"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766742    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766751    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766760    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766768    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766787    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy"
Jul 08 16:49:39.766824 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766795    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy"
Jul 08 16:49:39.767182 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766879    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kube-rbac-proxy"
Jul 08 16:49:39.767182 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766893    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="kserve-container"
Jul 08 16:49:39.767182 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.766905    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b92cda0-7ad8-44d0-8f33-eabcf52d8c90" containerName="agent"
Jul 08 16:49:39.770381 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.770362    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.772017 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.772000    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 08 16:49:39.772263 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.772249    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 08 16:49:39.779997 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.779976    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"]
Jul 08 16:49:39.807150 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.807126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:39.817652 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.817628    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"]
Jul 08 16:49:39.817868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.817845    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kserve-container" containerID="cri-o://3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47" gracePeriod=30
Jul 08 16:49:39.817868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.817857    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kube-rbac-proxy" containerID="cri-o://04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598" gracePeriod=30
Jul 08 16:49:39.876416 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.876380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:39.887991 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.887970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qb5k\" (UniqueName: \"kubernetes.io/projected/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-api-access-9qb5k\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.888091 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.888004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.888091 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.888027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.888168 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.888125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.888205 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.888163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f471518-d3c5-4e31-9f93-2a9235536b8e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.989239 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.989207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.989373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.989258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f471518-d3c5-4e31-9f93-2a9235536b8e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.989373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.989347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qb5k\" (UniqueName: \"kubernetes.io/projected/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-api-access-9qb5k\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.989373 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:49:39.989353    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 08 16:49:39.989533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.989393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.989533 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:49:39.989433    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls podName:7f471518-d3c5-4e31-9f93-2a9235536b8e nodeName:}" failed. No retries permitted until 2026-07-08 16:49:40.48941176 +0000 UTC m=+950.378826756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 08 16:49:39.989533 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.989472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.989733 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.989672    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f471518-d3c5-4e31-9f93-2a9235536b8e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.990135 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.990111    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.990247 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.990116    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:39.998856 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:39.998825    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qb5k\" (UniqueName: \"kubernetes.io/projected/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-api-access-9qb5k\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:40.060638 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.060606    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:49:40.090176 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.090151    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls\") pod \"b642155d-530a-4125-ab41-5d25bf7b6d30\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") "
Jul 08 16:49:40.090327 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.090243    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-rbac-proxy-sar-config\") pod \"b642155d-530a-4125-ab41-5d25bf7b6d30\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") "
Jul 08 16:49:40.090327 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.090270    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhtzk\" (UniqueName: \"kubernetes.io/projected/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-api-access-jhtzk\") pod \"b642155d-530a-4125-ab41-5d25bf7b6d30\" (UID: \"b642155d-530a-4125-ab41-5d25bf7b6d30\") "
Jul 08 16:49:40.090636 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.090610    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b642155d-530a-4125-ab41-5d25bf7b6d30" (UID: "b642155d-530a-4125-ab41-5d25bf7b6d30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:49:40.092241 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.092218    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b642155d-530a-4125-ab41-5d25bf7b6d30" (UID: "b642155d-530a-4125-ab41-5d25bf7b6d30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:49:40.092354 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.092269    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-api-access-jhtzk" (OuterVolumeSpecName: "kube-api-access-jhtzk") pod "b642155d-530a-4125-ab41-5d25bf7b6d30" (UID: "b642155d-530a-4125-ab41-5d25bf7b6d30"). InnerVolumeSpecName "kube-api-access-jhtzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:49:40.191068 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.191035    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtzk\" (UniqueName: \"kubernetes.io/projected/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-api-access-jhtzk\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:49:40.191068 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.191063    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b642155d-530a-4125-ab41-5d25bf7b6d30-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:49:40.191068 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.191073    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b642155d-530a-4125-ab41-5d25bf7b6d30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:49:40.400970 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.400888    2559 generic.go:356] "Generic (PLEG): container finished" podID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerID="40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5" exitCode=2
Jul 08 16:49:40.401147 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.400964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerDied","Data":"40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5"}
Jul 08 16:49:40.402398 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402374    2559 generic.go:356] "Generic (PLEG): container finished" podID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerID="04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598" exitCode=2
Jul 08 16:49:40.402398 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402395    2559 generic.go:356] "Generic (PLEG): container finished" podID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerID="3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47" exitCode=2
Jul 08 16:49:40.402549 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402420    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"
Jul 08 16:49:40.402549 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" event={"ID":"b642155d-530a-4125-ab41-5d25bf7b6d30","Type":"ContainerDied","Data":"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"}
Jul 08 16:49:40.402549 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" event={"ID":"b642155d-530a-4125-ab41-5d25bf7b6d30","Type":"ContainerDied","Data":"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"}
Jul 08 16:49:40.402549 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" event={"ID":"b642155d-530a-4125-ab41-5d25bf7b6d30","Type":"ContainerDied","Data":"c7f546af3c23b1fb1167d3810577c2d619bfb446ee7f950befced0e72bd16c2e"}
Jul 08 16:49:40.402549 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.402548    2559 scope.go:117] "RemoveContainer" containerID="04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"
Jul 08 16:49:40.412273 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.412255    2559 scope.go:117] "RemoveContainer" containerID="3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"
Jul 08 16:49:40.420557 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.420538    2559 scope.go:117] "RemoveContainer" containerID="04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"
Jul 08 16:49:40.420834 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:49:40.420818    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598\": container with ID starting with 04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598 not found: ID does not exist" containerID="04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"
Jul 08 16:49:40.420902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.420841    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"} err="failed to get container status \"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598\": rpc error: code = NotFound desc = could not find container \"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598\": container with ID starting with 04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598 not found: ID does not exist"
Jul 08 16:49:40.420902 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.420857    2559 scope.go:117] "RemoveContainer" containerID="3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"
Jul 08 16:49:40.421056 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:49:40.421041    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47\": container with ID starting with 3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47 not found: ID does not exist" containerID="3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"
Jul 08 16:49:40.421094 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.421062    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"} err="failed to get container status \"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47\": rpc error: code = NotFound desc = could not find container \"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47\": container with ID starting with 3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47 not found: ID does not exist"
Jul 08 16:49:40.421094 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.421075    2559 scope.go:117] "RemoveContainer" containerID="04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"
Jul 08 16:49:40.421317 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.421297    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598"} err="failed to get container status \"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598\": rpc error: code = NotFound desc = could not find container \"04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598\": container with ID starting with 04d8e9bef10040c29c9971b3a9db0bedc16d334124ca8ee0a634205359dba598 not found: ID does not exist"
Jul 08 16:49:40.421387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.421321    2559 scope.go:117] "RemoveContainer" containerID="3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"
Jul 08 16:49:40.421542 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.421527    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47"} err="failed to get container status \"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47\": rpc error: code = NotFound desc = could not find container \"3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47\": container with ID starting with 3ce9c860dd12196ab0abf415994be9dab5b69682bb5f52890a13c11a8dd78d47 not found: ID does not exist"
Jul 08 16:49:40.424327 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.424307    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"]
Jul 08 16:49:40.429600 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.429581    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb"]
Jul 08 16:49:40.494741 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.494707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:40.495179 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:49:40.494780    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 08 16:49:40.495179 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:49:40.494835    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls podName:7f471518-d3c5-4e31-9f93-2a9235536b8e nodeName:}" failed. No retries permitted until 2026-07-08 16:49:41.494821525 +0000 UTC m=+951.384236512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 08 16:49:40.671073 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.670972    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:49:40.671425 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.671399    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:40.673588 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.673559    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" path="/var/lib/kubelet/pods/b642155d-530a-4125-ab41-5d25bf7b6d30/volumes"
Jul 08 16:49:40.874691 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.874657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:40.989392 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:40.989352    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mcxtb" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": context deadline exceeded"
Jul 08 16:49:41.503193 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:41.503157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:41.505520 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:41.505491    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-v742l\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:41.580873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:41.580836    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:49:41.712989 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:41.712963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"]
Jul 08 16:49:41.715379 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:49:41.715350    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f471518_d3c5_4e31_9f93_2a9235536b8e.slice/crio-334f816c15750a6e8200a9c3dedd9bcda325fc172d7b37ebe5383b00470b4cba WatchSource:0}: Error finding container 334f816c15750a6e8200a9c3dedd9bcda325fc172d7b37ebe5383b00470b4cba: Status 404 returned error can't find the container with id 334f816c15750a6e8200a9c3dedd9bcda325fc172d7b37ebe5383b00470b4cba
Jul 08 16:49:42.412858 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:42.412822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerStarted","Data":"9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6"}
Jul 08 16:49:42.413021 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:42.412889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerStarted","Data":"334f816c15750a6e8200a9c3dedd9bcda325fc172d7b37ebe5383b00470b4cba"}
Jul 08 16:49:43.084240 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:43.084200    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 08 16:49:44.424229 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:44.424149    2559 generic.go:356] "Generic (PLEG): container finished" podID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerID="55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615" exitCode=0
Jul 08 16:49:44.424535 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:44.424227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerDied","Data":"55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615"}
Jul 08 16:49:48.084514 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:48.084473    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 08 16:49:50.670729 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:50.670675    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:49:50.671108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:50.671036    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:49:53.084472 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:53.084433    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 08 16:49:53.084857 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:53.084559    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:49:58.084127 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:49:58.084085    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 08 16:50:00.670847 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:00.670805    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 08 16:50:00.671295 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:00.671167    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:50:00.672880 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:00.672859    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:50:00.672968 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:00.672956    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:50:02.702968 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:02.702938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:03.084993 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:03.084951    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 08 16:50:08.084216 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:08.084170    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 08 16:50:09.905023 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:09.905001    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:50:10.032814 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.032778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls\") pod \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") "
Jul 08 16:50:10.032985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.032826    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-cabundle-cert\") pod \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") "
Jul 08 16:50:10.032985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.032850    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-rbac-proxy-sar-config\") pod \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") "
Jul 08 16:50:10.032985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.032872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpr68\" (UniqueName: \"kubernetes.io/projected/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-api-access-cpr68\") pod \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") "
Jul 08 16:50:10.032985 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.032908    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kserve-provision-location\") pod \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\" (UID: \"2af7ed25-87ce-42c2-928b-e69f0069b0c5\") "
Jul 08 16:50:10.033301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.033266    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2af7ed25-87ce-42c2-928b-e69f0069b0c5" (UID: "2af7ed25-87ce-42c2-928b-e69f0069b0c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:50:10.033301 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.033277    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2af7ed25-87ce-42c2-928b-e69f0069b0c5" (UID: "2af7ed25-87ce-42c2-928b-e69f0069b0c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:10.033467 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.033327    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2af7ed25-87ce-42c2-928b-e69f0069b0c5" (UID: "2af7ed25-87ce-42c2-928b-e69f0069b0c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:50:10.034950 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.034927    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2af7ed25-87ce-42c2-928b-e69f0069b0c5" (UID: "2af7ed25-87ce-42c2-928b-e69f0069b0c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:50:10.035022 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.034964    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-api-access-cpr68" (OuterVolumeSpecName: "kube-api-access-cpr68") pod "2af7ed25-87ce-42c2-928b-e69f0069b0c5" (UID: "2af7ed25-87ce-42c2-928b-e69f0069b0c5"). InnerVolumeSpecName "kube-api-access-cpr68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:50:10.134467 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.134436    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2af7ed25-87ce-42c2-928b-e69f0069b0c5-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:50:10.134467 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.134461    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:50:10.134604 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.134473    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:50:10.134604 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.134488    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpr68\" (UniqueName: \"kubernetes.io/projected/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kube-api-access-cpr68\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:50:10.134604 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.134500    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2af7ed25-87ce-42c2-928b-e69f0069b0c5-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:50:10.530223 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.530187    2559 generic.go:356] "Generic (PLEG): container finished" podID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerID="2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3" exitCode=137
Jul 08 16:50:10.530387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.530279    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"
Jul 08 16:50:10.530387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.530279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerDied","Data":"2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3"}
Jul 08 16:50:10.530387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.530373    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z" event={"ID":"2af7ed25-87ce-42c2-928b-e69f0069b0c5","Type":"ContainerDied","Data":"76c6adcc63e94ce6801eb56a53a795c778483dd0bddab14569351f04920b7323"}
Jul 08 16:50:10.530516 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.530404    2559 scope.go:117] "RemoveContainer" containerID="2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3"
Jul 08 16:50:10.538868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.538853    2559 scope.go:117] "RemoveContainer" containerID="40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5"
Jul 08 16:50:10.546907 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.546890    2559 scope.go:117] "RemoveContainer" containerID="55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615"
Jul 08 16:50:10.554837 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.554821    2559 scope.go:117] "RemoveContainer" containerID="c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194"
Jul 08 16:50:10.554908 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.554892    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"]
Jul 08 16:50:10.558834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.558815    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5d78f49685-sx85z"]
Jul 08 16:50:10.561795 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.561778    2559 scope.go:117] "RemoveContainer" containerID="2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3"
Jul 08 16:50:10.562024 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:50:10.562005    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3\": container with ID starting with 2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3 not found: ID does not exist" containerID="2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3"
Jul 08 16:50:10.562076 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562035    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3"} err="failed to get container status \"2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3\": rpc error: code = NotFound desc = could not find container \"2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3\": container with ID starting with 2d3da3f495b8f13f62469ded40180e3b71b48324ae55fe317ecd1c3ec8cf35d3 not found: ID does not exist"
Jul 08 16:50:10.562076 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562058    2559 scope.go:117] "RemoveContainer" containerID="40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5"
Jul 08 16:50:10.562319 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:50:10.562304    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5\": container with ID starting with 40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5 not found: ID does not exist" containerID="40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5"
Jul 08 16:50:10.562366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562323    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5"} err="failed to get container status \"40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5\": rpc error: code = NotFound desc = could not find container \"40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5\": container with ID starting with 40aff49285c81f58719f138f529490f293edd8b1970c9de6307242c0ffdac4d5 not found: ID does not exist"
Jul 08 16:50:10.562366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562338    2559 scope.go:117] "RemoveContainer" containerID="55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615"
Jul 08 16:50:10.562547 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:50:10.562530    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615\": container with ID starting with 55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615 not found: ID does not exist" containerID="55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615"
Jul 08 16:50:10.562590 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562560    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615"} err="failed to get container status \"55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615\": rpc error: code = NotFound desc = could not find container \"55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615\": container with ID starting with 55c2411df8f9e22daa5f177d6d820199048685fa2ab44b3b41df24e3aab6c615 not found: ID does not exist"
Jul 08 16:50:10.562590 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562574    2559 scope.go:117] "RemoveContainer" containerID="c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194"
Jul 08 16:50:10.562813 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:50:10.562798    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194\": container with ID starting with c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194 not found: ID does not exist" containerID="c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194"
Jul 08 16:50:10.562865 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.562818    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194"} err="failed to get container status \"c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194\": rpc error: code = NotFound desc = could not find container \"c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194\": container with ID starting with c4f334626fe0b906a141d3722695f4e9fa38d4b60fc78900d6653c747df55194 not found: ID does not exist"
Jul 08 16:50:10.672052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:10.672030    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" path="/var/lib/kubelet/pods/2af7ed25-87ce-42c2-928b-e69f0069b0c5/volumes"
Jul 08 16:50:11.535207 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:11.535127    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerID="9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6" exitCode=0
Jul 08 16:50:11.535608 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:11.535201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerDied","Data":"9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6"}
Jul 08 16:50:59.903094 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:50:59.903054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:22.097622 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:22.097585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerStarted","Data":"dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd"}
Jul 08 16:52:22.098083 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:22.097672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerStarted","Data":"0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2"}
Jul 08 16:52:22.098083 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:22.097812    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:52:22.098083 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:22.097869    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:52:22.099318 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:22.099292    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 08 16:52:22.130562 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:22.130502    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" podStartSLOduration=3.972134829 podStartE2EDuration="2m43.1304835s" totalImagesPullingTime="2m9.685619329s" totalInitContainerRuntime="29.472729342s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:52:22.128086226 +0000 UTC m=+1112.017501236" watchObservedRunningTime="2026-07-08 16:52:22.1304835 +0000 UTC m=+1112.019898511"
Jul 08 16:52:23.101977 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:23.101932    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 08 16:52:28.107014 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:28.106985    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:52:28.107650 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:28.107621    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 08 16:52:38.108709 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:38.108666    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:52:39.964638 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:39.964601    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"]
Jul 08 16:52:39.965143 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:39.964829    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container" containerID="cri-o://0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2" gracePeriod=30
Jul 08 16:52:39.965143 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:39.964874    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kube-rbac-proxy" containerID="cri-o://dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd" gracePeriod=30
Jul 08 16:52:40.039726 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.039693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:40.169893 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.169860    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerID="dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd" exitCode=2
Jul 08 16:52:40.170043 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.169896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerDied","Data":"dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd"}
Jul 08 16:52:40.754319 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.754294    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:52:40.866543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.866475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-rbac-proxy-sar-config\") pod \"7f471518-d3c5-4e31-9f93-2a9235536b8e\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") "
Jul 08 16:52:40.866543 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.866519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") pod \"7f471518-d3c5-4e31-9f93-2a9235536b8e\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") "
Jul 08 16:52:40.866763 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.866554    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f471518-d3c5-4e31-9f93-2a9235536b8e-kserve-provision-location\") pod \"7f471518-d3c5-4e31-9f93-2a9235536b8e\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") "
Jul 08 16:52:40.866763 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.866634    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-cabundle-cert\") pod \"7f471518-d3c5-4e31-9f93-2a9235536b8e\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") "
Jul 08 16:52:40.866763 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.866667    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qb5k\" (UniqueName: \"kubernetes.io/projected/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-api-access-9qb5k\") pod \"7f471518-d3c5-4e31-9f93-2a9235536b8e\" (UID: \"7f471518-d3c5-4e31-9f93-2a9235536b8e\") "
Jul 08 16:52:40.866912 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.866799    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f471518-d3c5-4e31-9f93-2a9235536b8e" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:52:40.867028 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.867000    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f471518-d3c5-4e31-9f93-2a9235536b8e" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:52:40.868705 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.868672    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f471518-d3c5-4e31-9f93-2a9235536b8e" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:52:40.868804 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.868783    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-api-access-9qb5k" (OuterVolumeSpecName: "kube-api-access-9qb5k") pod "7f471518-d3c5-4e31-9f93-2a9235536b8e" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e"). InnerVolumeSpecName "kube-api-access-9qb5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:52:40.871338 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.871318    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f471518-d3c5-4e31-9f93-2a9235536b8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f471518-d3c5-4e31-9f93-2a9235536b8e" (UID: "7f471518-d3c5-4e31-9f93-2a9235536b8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:40.967365 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.967341    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f471518-d3c5-4e31-9f93-2a9235536b8e-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.967365 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.967362    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.967705 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.967376    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qb5k\" (UniqueName: \"kubernetes.io/projected/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-api-access-9qb5k\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.967705 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.967386    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f471518-d3c5-4e31-9f93-2a9235536b8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:52:40.967705 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:40.967395    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f471518-d3c5-4e31-9f93-2a9235536b8e-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:52:41.175725 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.175654    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerID="0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2" exitCode=0
Jul 08 16:52:41.175849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.175715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerDied","Data":"0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2"}
Jul 08 16:52:41.175849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.175735    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"
Jul 08 16:52:41.175849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.175785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l" event={"ID":"7f471518-d3c5-4e31-9f93-2a9235536b8e","Type":"ContainerDied","Data":"334f816c15750a6e8200a9c3dedd9bcda325fc172d7b37ebe5383b00470b4cba"}
Jul 08 16:52:41.175849 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.175817    2559 scope.go:117] "RemoveContainer" containerID="dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd"
Jul 08 16:52:41.184388 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.184373    2559 scope.go:117] "RemoveContainer" containerID="0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2"
Jul 08 16:52:41.191854 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.191826    2559 scope.go:117] "RemoveContainer" containerID="9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6"
Jul 08 16:52:41.198905 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.198890    2559 scope.go:117] "RemoveContainer" containerID="dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd"
Jul 08 16:52:41.199137 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:52:41.199119    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd\": container with ID starting with dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd not found: ID does not exist" containerID="dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd"
Jul 08 16:52:41.199184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.199144    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd"} err="failed to get container status \"dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd\": rpc error: code = NotFound desc = could not find container \"dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd\": container with ID starting with dd48e5eacdc7e6eec090be73afc41451938f92c8afffdf1c3dc213dcc178a0dd not found: ID does not exist"
Jul 08 16:52:41.199184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.199160    2559 scope.go:117] "RemoveContainer" containerID="0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2"
Jul 08 16:52:41.199366 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:52:41.199350    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2\": container with ID starting with 0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2 not found: ID does not exist" containerID="0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2"
Jul 08 16:52:41.199402 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.199371    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2"} err="failed to get container status \"0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2\": rpc error: code = NotFound desc = could not find container \"0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2\": container with ID starting with 0306d078b4accc7c2e2ceeb3ceb588777ef97c8833cbe42db5e389355024bce2 not found: ID does not exist"
Jul 08 16:52:41.199402 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.199386    2559 scope.go:117] "RemoveContainer" containerID="9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6"
Jul 08 16:52:41.199570 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:52:41.199555    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6\": container with ID starting with 9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6 not found: ID does not exist" containerID="9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6"
Jul 08 16:52:41.199613 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.199574    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6"} err="failed to get container status \"9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6\": rpc error: code = NotFound desc = could not find container \"9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6\": container with ID starting with 9e2bffb5689d32512dfe2ba523290b8976150400f254234e41fb96603e8116d6 not found: ID does not exist"
Jul 08 16:52:41.203086 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.203067    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"]
Jul 08 16:52:41.207606 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:41.207579    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-v742l"]
Jul 08 16:52:42.672767 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:42.672730    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" path="/var/lib/kubelet/pods/7f471518-d3c5-4e31-9f93-2a9235536b8e/volumes"
Jul 08 16:52:45.396389 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.396356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:45.461943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.461914    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"]
Jul 08 16:52:45.462205 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462192    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462250 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462207    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462250 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462220    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="storage-initializer"
Jul 08 16:52:45.462250 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462226    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="storage-initializer"
Jul 08 16:52:45.462250 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462237    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container"
Jul 08 16:52:45.462250 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462242    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462254    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462263    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462270    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462275    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462285    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462290    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462297    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="storage-initializer"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462302    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="storage-initializer"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462308    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462313    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462318    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462324    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462376    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462386    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462393    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kserve-container"
Jul 08 16:52:45.462399 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462398    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b642155d-530a-4125-ab41-5d25bf7b6d30" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462405    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="kube-rbac-proxy"
Jul 08 16:52:45.462899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462416    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af7ed25-87ce-42c2-928b-e69f0069b0c5" containerName="agent"
Jul 08 16:52:45.462899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.462423    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f471518-d3c5-4e31-9f93-2a9235536b8e" containerName="kserve-container"
Jul 08 16:52:45.486014 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.485986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"]
Jul 08 16:52:45.486121 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.486089    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.488103 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488076    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 08 16:52:45.488217 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488174    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 08 16:52:45.488217 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488191    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:52:45.488358 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488249    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:52:45.488358 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488260    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-924bj\""
Jul 08 16:52:45.488480 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488468    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:52:45.488525 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.488500    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:52:45.598128 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.598107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-api-access-6k8lk\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.598238 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.598135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.598238 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.598169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f51c1734-7b45-46de-a51d-0ff6945c850c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.598238 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.598194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.598344 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.598260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f51c1734-7b45-46de-a51d-0ff6945c850c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.650880 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.646607    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:45.698917 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.698890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-api-access-6k8lk\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699048 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.698921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699048 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.698952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f51c1734-7b45-46de-a51d-0ff6945c850c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699048 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.698971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699048 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.699014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f51c1734-7b45-46de-a51d-0ff6945c850c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699458 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.699433    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f51c1734-7b45-46de-a51d-0ff6945c850c-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699747 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.699728    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.699834 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.699729    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.701431 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.701411    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f51c1734-7b45-46de-a51d-0ff6945c850c-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.708169 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.708150    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-api-access-6k8lk\") pod \"isvc-autogluon-v2-predictor-67b849d456-jggmp\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.796165 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.796145    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:52:45.935316 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:45.935214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"]
Jul 08 16:52:45.937601 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:52:45.937575    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51c1734_7b45_46de_a51d_0ff6945c850c.slice/crio-19f946854c4ba9d6f24eb73da7330a7590008dfa654decb83fea721072816d69 WatchSource:0}: Error finding container 19f946854c4ba9d6f24eb73da7330a7590008dfa654decb83fea721072816d69: Status 404 returned error can't find the container with id 19f946854c4ba9d6f24eb73da7330a7590008dfa654decb83fea721072816d69
Jul 08 16:52:46.197303 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:46.197218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerStarted","Data":"6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9"}
Jul 08 16:52:46.197303 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:52:46.197282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerStarted","Data":"19f946854c4ba9d6f24eb73da7330a7590008dfa654decb83fea721072816d69"}
Jul 08 16:53:17.325877 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:17.325840    2559 generic.go:356] "Generic (PLEG): container finished" podID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerID="6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9" exitCode=0
Jul 08 16:53:17.326303 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:17.325917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerDied","Data":"6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9"}
Jul 08 16:53:18.335159 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:18.335122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerStarted","Data":"99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4"}
Jul 08 16:53:18.335534 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:18.335189    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerStarted","Data":"6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c"}
Jul 08 16:53:18.335534 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:18.335379    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:53:18.354724 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:18.354653    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" podStartSLOduration=2.699074062 podStartE2EDuration="33.354633533s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.655559471s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:53:18.353548298 +0000 UTC m=+1168.242963309" watchObservedRunningTime="2026-07-08 16:53:18.354633533 +0000 UTC m=+1168.244048545"
Jul 08 16:53:19.340127 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:19.340095    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:53:25.349226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:25.349193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:53:50.617848 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:50.617820    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:53:50.621197 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:53:50.621173    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:54:48.673003 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:48.672916    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:54:55.600826 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:55.600793    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"]
Jul 08 16:54:55.601198 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:55.601028    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kserve-container" containerID="cri-o://6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c" gracePeriod=30
Jul 08 16:54:55.601198 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:55.601071    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kube-rbac-proxy" containerID="cri-o://99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4" gracePeriod=30
Jul 08 16:54:55.654668 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:55.654644    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:55.718137 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:55.718116    2559 generic.go:356] "Generic (PLEG): container finished" podID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerID="99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4" exitCode=2
Jul 08 16:54:55.718228 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:55.718172    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerDied","Data":"99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4"}
Jul 08 16:54:56.438861 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.438839    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:54:56.448753 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.448735    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f51c1734-7b45-46de-a51d-0ff6945c850c-proxy-tls\") pod \"f51c1734-7b45-46de-a51d-0ff6945c850c\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") "
Jul 08 16:54:56.448873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.448766    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-rbac-proxy-sar-config\") pod \"f51c1734-7b45-46de-a51d-0ff6945c850c\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") "
Jul 08 16:54:56.448873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.448796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f51c1734-7b45-46de-a51d-0ff6945c850c-kserve-provision-location\") pod \"f51c1734-7b45-46de-a51d-0ff6945c850c\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") "
Jul 08 16:54:56.448873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.448824    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-api-access-6k8lk\") pod \"f51c1734-7b45-46de-a51d-0ff6945c850c\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") "
Jul 08 16:54:56.448873 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.448848    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-cabundle-cert\") pod \"f51c1734-7b45-46de-a51d-0ff6945c850c\" (UID: \"f51c1734-7b45-46de-a51d-0ff6945c850c\") "
Jul 08 16:54:56.449202 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.449170    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f51c1734-7b45-46de-a51d-0ff6945c850c" (UID: "f51c1734-7b45-46de-a51d-0ff6945c850c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:56.449286 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.449260    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f51c1734-7b45-46de-a51d-0ff6945c850c" (UID: "f51c1734-7b45-46de-a51d-0ff6945c850c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:56.450693 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.450658    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51c1734-7b45-46de-a51d-0ff6945c850c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f51c1734-7b45-46de-a51d-0ff6945c850c" (UID: "f51c1734-7b45-46de-a51d-0ff6945c850c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:54:56.450864 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.450844    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-api-access-6k8lk" (OuterVolumeSpecName: "kube-api-access-6k8lk") pod "f51c1734-7b45-46de-a51d-0ff6945c850c" (UID: "f51c1734-7b45-46de-a51d-0ff6945c850c"). InnerVolumeSpecName "kube-api-access-6k8lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:54:56.453025 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.453001    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51c1734-7b45-46de-a51d-0ff6945c850c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f51c1734-7b45-46de-a51d-0ff6945c850c" (UID: "f51c1734-7b45-46de-a51d-0ff6945c850c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:54:56.550053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.549999    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f51c1734-7b45-46de-a51d-0ff6945c850c-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:54:56.550053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.550021    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8lk\" (UniqueName: \"kubernetes.io/projected/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-api-access-6k8lk\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:54:56.550053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.550031    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:54:56.550053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.550040    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f51c1734-7b45-46de-a51d-0ff6945c850c-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:54:56.550053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.550047    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f51c1734-7b45-46de-a51d-0ff6945c850c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:54:56.723427 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.723395    2559 generic.go:356] "Generic (PLEG): container finished" podID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerID="6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c" exitCode=0
Jul 08 16:54:56.723766 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.723440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerDied","Data":"6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c"}
Jul 08 16:54:56.723766 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.723462    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"
Jul 08 16:54:56.723766 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.723524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp" event={"ID":"f51c1734-7b45-46de-a51d-0ff6945c850c","Type":"ContainerDied","Data":"19f946854c4ba9d6f24eb73da7330a7590008dfa654decb83fea721072816d69"}
Jul 08 16:54:56.723766 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.723554    2559 scope.go:117] "RemoveContainer" containerID="99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4"
Jul 08 16:54:56.731691 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.731666    2559 scope.go:117] "RemoveContainer" containerID="6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c"
Jul 08 16:54:56.738743 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.738728    2559 scope.go:117] "RemoveContainer" containerID="6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9"
Jul 08 16:54:56.745256 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.745218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"]
Jul 08 16:54:56.746002 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.745990    2559 scope.go:117] "RemoveContainer" containerID="99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4"
Jul 08 16:54:56.746246 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:54:56.746228    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4\": container with ID starting with 99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4 not found: ID does not exist" containerID="99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4"
Jul 08 16:54:56.746293 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.746257    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4"} err="failed to get container status \"99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4\": rpc error: code = NotFound desc = could not find container \"99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4\": container with ID starting with 99de1fb720440eadc63648a29fdc3e2030c4aa2d103e49ca19754581fcfd8ce4 not found: ID does not exist"
Jul 08 16:54:56.746293 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.746275    2559 scope.go:117] "RemoveContainer" containerID="6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c"
Jul 08 16:54:56.746480 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:54:56.746464    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c\": container with ID starting with 6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c not found: ID does not exist" containerID="6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c"
Jul 08 16:54:56.746538 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.746485    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c"} err="failed to get container status \"6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c\": rpc error: code = NotFound desc = could not find container \"6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c\": container with ID starting with 6b86c9662825c7641b47b50bd94f4a831ec57b16d1ddb90a3c1fc99963b6e16c not found: ID does not exist"
Jul 08 16:54:56.746538 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.746499    2559 scope.go:117] "RemoveContainer" containerID="6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9"
Jul 08 16:54:56.746699 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:54:56.746671    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9\": container with ID starting with 6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9 not found: ID does not exist" containerID="6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9"
Jul 08 16:54:56.746745 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.746704    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9"} err="failed to get container status \"6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9\": rpc error: code = NotFound desc = could not find container \"6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9\": container with ID starting with 6c1c432512725c67e4806b4be458d6a4b9e6e8416ba370f6ff98ccb3169f48b9 not found: ID does not exist"
Jul 08 16:54:56.750103 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:56.750083    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jggmp"]
Jul 08 16:54:58.673177 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:54:58.673144    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" path="/var/lib/kubelet/pods/f51c1734-7b45-46de-a51d-0ff6945c850c/volumes"
Jul 08 16:55:00.749808 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.749774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:00.811049 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811023    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"]
Jul 08 16:55:00.811311 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811299    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="storage-initializer"
Jul 08 16:55:00.811351 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811314    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="storage-initializer"
Jul 08 16:55:00.811351 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811325    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kube-rbac-proxy"
Jul 08 16:55:00.811351 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811331    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kube-rbac-proxy"
Jul 08 16:55:00.811351 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811346    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kserve-container"
Jul 08 16:55:00.811482 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kserve-container"
Jul 08 16:55:00.811482 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811396    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kserve-container"
Jul 08 16:55:00.811482 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.811403    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51c1734-7b45-46de-a51d-0ff6945c850c" containerName="kube-rbac-proxy"
Jul 08 16:55:00.815646 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.815629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.817409 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.817386    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:55:00.817529 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.817513    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:55:00.817586 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.817515    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 08 16:55:00.817983 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.817966    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-924bj\""
Jul 08 16:55:00.818035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.818017    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:55:00.818035 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.818030    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:55:00.818131 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.818017    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 08 16:55:00.824476 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.824446    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"]
Jul 08 16:55:00.876883 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.876860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:00.877180 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.877162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.877252 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.877219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.877295 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.877272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9pg9\" (UniqueName: \"kubernetes.io/projected/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-api-access-r9pg9\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.877334 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.877299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.877369 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.877335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978075 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9pg9\" (UniqueName: \"kubernetes.io/projected/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-api-access-r9pg9\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978184 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978556 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978538    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978740 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978721    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.978832 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.978809    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.980200 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.980183    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:00.987064 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:00.987045    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9pg9\" (UniqueName: \"kubernetes.io/projected/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-api-access-r9pg9\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:01.125930 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:01.125867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:01.252042 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:01.252014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"]
Jul 08 16:55:01.255234 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:55:01.255208    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa2ee69_27f3_415c_95e0_16d7e6e4fb6c.slice/crio-23fe9b57d00b952cfa9b07c0e9b208ef33cbe64e87b492604f2bf52d2d88556d WatchSource:0}: Error finding container 23fe9b57d00b952cfa9b07c0e9b208ef33cbe64e87b492604f2bf52d2d88556d: Status 404 returned error can't find the container with id 23fe9b57d00b952cfa9b07c0e9b208ef33cbe64e87b492604f2bf52d2d88556d
Jul 08 16:55:01.744950 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:01.744914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerStarted","Data":"e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7"}
Jul 08 16:55:01.745123 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:01.744971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerStarted","Data":"23fe9b57d00b952cfa9b07c0e9b208ef33cbe64e87b492604f2bf52d2d88556d"}
Jul 08 16:55:06.765309 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:06.765276    2559 generic.go:356] "Generic (PLEG): container finished" podID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerID="e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7" exitCode=0
Jul 08 16:55:06.765798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:06.765355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerDied","Data":"e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7"}
Jul 08 16:55:07.770960 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:07.770926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerStarted","Data":"dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc"}
Jul 08 16:55:07.771366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:07.770987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerStarted","Data":"9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013"}
Jul 08 16:55:07.771366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:07.771206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:07.771366 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:07.771254    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:07.790772 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:07.790724    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" podStartSLOduration=2.975215636 podStartE2EDuration="7.790707684s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.815492048s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:07.789411855 +0000 UTC m=+1277.678826864" watchObservedRunningTime="2026-07-08 16:55:07.790707684 +0000 UTC m=+1277.680122701"
Jul 08 16:55:13.780810 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:13.780775    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:55:48.899210 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:55:48.899176    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:43.672777 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:43.672749    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:56:50.937441 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:50.937403    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"]
Jul 08 16:56:50.937894 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:50.937648    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kserve-container" containerID="cri-o://9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013" gracePeriod=30
Jul 08 16:56:50.937894 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:50.937738    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kube-rbac-proxy" containerID="cri-o://dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc" gracePeriod=30
Jul 08 16:56:50.959408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:50.959386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:51.164946 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.164905    2559 generic.go:356] "Generic (PLEG): container finished" podID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerID="dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc" exitCode=2
Jul 08 16:56:51.165109 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.164967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerDied","Data":"dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc"}
Jul 08 16:56:51.777463 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.777440    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:56:51.878349 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-cabundle-cert\") pod \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") "
Jul 08 16:56:51.878349 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878308    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-proxy-tls\") pod \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") "
Jul 08 16:56:51.878349 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878334    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kserve-provision-location\") pod \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") "
Jul 08 16:56:51.878617 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-rbac-proxy-sar-config\") pod \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") "
Jul 08 16:56:51.878617 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878494    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9pg9\" (UniqueName: \"kubernetes.io/projected/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-api-access-r9pg9\") pod \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\" (UID: \"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c\") "
Jul 08 16:56:51.878767 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878720    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" (UID: "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:56:51.878767 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.878752    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" (UID: "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:56:51.880408 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.880380    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" (UID: "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:56:51.880515 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.880468    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-api-access-r9pg9" (OuterVolumeSpecName: "kube-api-access-r9pg9") pod "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" (UID: "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c"). InnerVolumeSpecName "kube-api-access-r9pg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:56:51.884929 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.884904    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" (UID: "caa2ee69-27f3-415c-95e0-16d7e6e4fb6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:56:51.979590 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.979563    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:56:51.979590 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.979586    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:56:51.979975 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.979597    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9pg9\" (UniqueName: \"kubernetes.io/projected/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-kube-api-access-r9pg9\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:56:51.979975 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.979611    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:56:51.979975 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:51.979620    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:56:52.170551 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.170480    2559 generic.go:356] "Generic (PLEG): container finished" podID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerID="9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013" exitCode=0
Jul 08 16:56:52.170551 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.170513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerDied","Data":"9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013"}
Jul 08 16:56:52.170551 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.170532    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"
Jul 08 16:56:52.170817 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.170560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb" event={"ID":"caa2ee69-27f3-415c-95e0-16d7e6e4fb6c","Type":"ContainerDied","Data":"23fe9b57d00b952cfa9b07c0e9b208ef33cbe64e87b492604f2bf52d2d88556d"}
Jul 08 16:56:52.170817 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.170589    2559 scope.go:117] "RemoveContainer" containerID="dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc"
Jul 08 16:56:52.183420 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.183391    2559 scope.go:117] "RemoveContainer" containerID="9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013"
Jul 08 16:56:52.190623 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.190605    2559 scope.go:117] "RemoveContainer" containerID="e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7"
Jul 08 16:56:52.197421 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.197406    2559 scope.go:117] "RemoveContainer" containerID="dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc"
Jul 08 16:56:52.197643 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:56:52.197625    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc\": container with ID starting with dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc not found: ID does not exist" containerID="dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc"
Jul 08 16:56:52.197711 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.197650    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc"} err="failed to get container status \"dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc\": rpc error: code = NotFound desc = could not find container \"dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc\": container with ID starting with dd6d5732d140c507ff55ef125a1d68ec36d472b221d67e50d45e090bd9354bfc not found: ID does not exist"
Jul 08 16:56:52.197711 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.197667    2559 scope.go:117] "RemoveContainer" containerID="9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013"
Jul 08 16:56:52.197910 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:56:52.197896    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013\": container with ID starting with 9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013 not found: ID does not exist" containerID="9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013"
Jul 08 16:56:52.197950 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.197914    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013"} err="failed to get container status \"9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013\": rpc error: code = NotFound desc = could not find container \"9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013\": container with ID starting with 9c65d7a73d8855fa8442a8c650ea7fa2b05277c2b3da05e22605e77401d19013 not found: ID does not exist"
Jul 08 16:56:52.197950 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.197926    2559 scope.go:117] "RemoveContainer" containerID="e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7"
Jul 08 16:56:52.198169 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:56:52.198144    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7\": container with ID starting with e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7 not found: ID does not exist" containerID="e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7"
Jul 08 16:56:52.198329 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.198175    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7"} err="failed to get container status \"e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7\": rpc error: code = NotFound desc = could not find container \"e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7\": container with ID starting with e6a6d2758650481d41fa26ab9775df97df2c55dedc300da0ddd74b1f557608a7 not found: ID does not exist"
Jul 08 16:56:52.200393 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.200376    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"]
Jul 08 16:56:52.204861 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.204841    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-q7ctb"]
Jul 08 16:56:52.672352 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:52.672322    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" path="/var/lib/kubelet/pods/caa2ee69-27f3-415c-95e0-16d7e6e4fb6c/volumes"
Jul 08 16:56:56.121508 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.121459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:56.140173 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140145    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"]
Jul 08 16:56:56.140420 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140409    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="storage-initializer"
Jul 08 16:56:56.140463 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140421    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="storage-initializer"
Jul 08 16:56:56.140463 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140435    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kserve-container"
Jul 08 16:56:56.140463 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140441    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kserve-container"
Jul 08 16:56:56.140463 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140448    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kube-rbac-proxy"
Jul 08 16:56:56.140463 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140454    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kube-rbac-proxy"
Jul 08 16:56:56.140609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140500    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kserve-container"
Jul 08 16:56:56.140609 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.140507    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa2ee69-27f3-415c-95e0-16d7e6e4fb6c" containerName="kube-rbac-proxy"
Jul 08 16:56:56.145216 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.145199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.148285 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.148254    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 08 16:56:56.148393 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.148302    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:56:56.148393 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.148321    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:56:56.149117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.149091    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 08 16:56:56.149117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.149100    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:56:56.149117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.149110    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:56:56.149319 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.149093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-924bj\""
Jul 08 16:56:56.157450 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.157426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"]
Jul 08 16:56:56.208943 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.208918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.209075 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.208955    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.209075 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.208973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989097cc-9869-4539-a82d-f7acf3d2da14-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.209075 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.208991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfqr\" (UniqueName: \"kubernetes.io/projected/989097cc-9869-4539-a82d-f7acf3d2da14-kube-api-access-cgfqr\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.209229 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.209121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/989097cc-9869-4539-a82d-f7acf3d2da14-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.263083 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.263056    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:56.310287 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/989097cc-9869-4539-a82d-f7acf3d2da14-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.310379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.310379 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.310468 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310436    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989097cc-9869-4539-a82d-f7acf3d2da14-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.310524 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfqr\" (UniqueName: \"kubernetes.io/projected/989097cc-9869-4539-a82d-f7acf3d2da14-kube-api-access-cgfqr\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.310725 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310703    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/989097cc-9869-4539-a82d-f7acf3d2da14-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.311006 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.310983    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.311126 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.311036    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.312764 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.312739    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989097cc-9869-4539-a82d-f7acf3d2da14-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.320249 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.320228    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfqr\" (UniqueName: \"kubernetes.io/projected/989097cc-9869-4539-a82d-f7acf3d2da14-kube-api-access-cgfqr\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.454726 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.454657    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:56:56.584329 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:56.584284    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"]
Jul 08 16:56:56.586694 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:56:56.586656    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989097cc_9869_4539_a82d_f7acf3d2da14.slice/crio-0059aa6f92a39412192801ff85cfe593cdcfa98c4960295276ffaa306071899c WatchSource:0}: Error finding container 0059aa6f92a39412192801ff85cfe593cdcfa98c4960295276ffaa306071899c: Status 404 returned error can't find the container with id 0059aa6f92a39412192801ff85cfe593cdcfa98c4960295276ffaa306071899c
Jul 08 16:56:57.193488 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:57.193449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerStarted","Data":"bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084"}
Jul 08 16:56:57.193916 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:56:57.193553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerStarted","Data":"0059aa6f92a39412192801ff85cfe593cdcfa98c4960295276ffaa306071899c"}
Jul 08 16:57:02.216226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:02.216193    2559 generic.go:356] "Generic (PLEG): container finished" podID="989097cc-9869-4539-a82d-f7acf3d2da14" containerID="bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084" exitCode=0
Jul 08 16:57:02.216226 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:02.216232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerDied","Data":"bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084"}
Jul 08 16:57:03.222883 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:03.222852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerStarted","Data":"6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df"}
Jul 08 16:57:03.223293 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:03.222908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerStarted","Data":"21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea"}
Jul 08 16:57:03.223293 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:03.223106    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:57:03.223293 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:03.223177    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:57:09.233140 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:09.233113    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:57:09.256647 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:57:09.256596    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" podStartSLOduration=8.669863432 podStartE2EDuration="13.256581851s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.586718419s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:57:03.24239009 +0000 UTC m=+1393.131805136" watchObservedRunningTime="2026-07-08 16:57:09.256581851 +0000 UTC m=+1399.145996862"
Jul 08 16:58:40.678053 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:40.675974    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:58:46.277405 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:46.277373    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"]
Jul 08 16:58:46.277862 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:46.277596    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kserve-container" containerID="cri-o://21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea" gracePeriod=30
Jul 08 16:58:46.277862 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:46.277639    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kube-rbac-proxy" containerID="cri-o://6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df" gracePeriod=30
Jul 08 16:58:46.337556 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:46.337524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:46.624635 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:46.624555    2559 generic.go:356] "Generic (PLEG): container finished" podID="989097cc-9869-4539-a82d-f7acf3d2da14" containerID="6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df" exitCode=2
Jul 08 16:58:46.624635 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:46.624592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerDied","Data":"6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df"}
Jul 08 16:58:47.015565 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.015535    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:58:47.115674 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.115648    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989097cc-9869-4539-a82d-f7acf3d2da14-proxy-tls\") pod \"989097cc-9869-4539-a82d-f7acf3d2da14\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") "
Jul 08 16:58:47.115846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.115699    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-cabundle-cert\") pod \"989097cc-9869-4539-a82d-f7acf3d2da14\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") "
Jul 08 16:58:47.115846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.115748    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-kube-rbac-proxy-sar-config\") pod \"989097cc-9869-4539-a82d-f7acf3d2da14\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") "
Jul 08 16:58:47.115846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.115768    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgfqr\" (UniqueName: \"kubernetes.io/projected/989097cc-9869-4539-a82d-f7acf3d2da14-kube-api-access-cgfqr\") pod \"989097cc-9869-4539-a82d-f7acf3d2da14\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") "
Jul 08 16:58:47.115846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.115783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/989097cc-9869-4539-a82d-f7acf3d2da14-kserve-provision-location\") pod \"989097cc-9869-4539-a82d-f7acf3d2da14\" (UID: \"989097cc-9869-4539-a82d-f7acf3d2da14\") "
Jul 08 16:58:47.116108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.116078    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "989097cc-9869-4539-a82d-f7acf3d2da14" (UID: "989097cc-9869-4539-a82d-f7acf3d2da14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:58:47.116108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.116091    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "989097cc-9869-4539-a82d-f7acf3d2da14" (UID: "989097cc-9869-4539-a82d-f7acf3d2da14"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:58:47.117754 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.117721    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989097cc-9869-4539-a82d-f7acf3d2da14-kube-api-access-cgfqr" (OuterVolumeSpecName: "kube-api-access-cgfqr") pod "989097cc-9869-4539-a82d-f7acf3d2da14" (UID: "989097cc-9869-4539-a82d-f7acf3d2da14"). InnerVolumeSpecName "kube-api-access-cgfqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:58:47.117754 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.117744    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989097cc-9869-4539-a82d-f7acf3d2da14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "989097cc-9869-4539-a82d-f7acf3d2da14" (UID: "989097cc-9869-4539-a82d-f7acf3d2da14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:58:47.120276 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.120254    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989097cc-9869-4539-a82d-f7acf3d2da14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "989097cc-9869-4539-a82d-f7acf3d2da14" (UID: "989097cc-9869-4539-a82d-f7acf3d2da14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:58:47.216267 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.216205    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:58:47.216267 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.216228    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfqr\" (UniqueName: \"kubernetes.io/projected/989097cc-9869-4539-a82d-f7acf3d2da14-kube-api-access-cgfqr\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:58:47.216267 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.216238    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/989097cc-9869-4539-a82d-f7acf3d2da14-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:58:47.216267 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.216248    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989097cc-9869-4539-a82d-f7acf3d2da14-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:58:47.216267 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.216256    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/989097cc-9869-4539-a82d-f7acf3d2da14-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:58:47.630199 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.630163    2559 generic.go:356] "Generic (PLEG): container finished" podID="989097cc-9869-4539-a82d-f7acf3d2da14" containerID="21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea" exitCode=0
Jul 08 16:58:47.630608 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.630205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerDied","Data":"21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea"}
Jul 08 16:58:47.630608 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.630230    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"
Jul 08 16:58:47.630608 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.630266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb" event={"ID":"989097cc-9869-4539-a82d-f7acf3d2da14","Type":"ContainerDied","Data":"0059aa6f92a39412192801ff85cfe593cdcfa98c4960295276ffaa306071899c"}
Jul 08 16:58:47.630608 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.630303    2559 scope.go:117] "RemoveContainer" containerID="6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df"
Jul 08 16:58:47.642692 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.642661    2559 scope.go:117] "RemoveContainer" containerID="21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea"
Jul 08 16:58:47.649990 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.649973    2559 scope.go:117] "RemoveContainer" containerID="bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084"
Jul 08 16:58:47.656638 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.656618    2559 scope.go:117] "RemoveContainer" containerID="6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df"
Jul 08 16:58:47.657002 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:58:47.656967    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df\": container with ID starting with 6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df not found: ID does not exist" containerID="6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df"
Jul 08 16:58:47.657117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.657013    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df"} err="failed to get container status \"6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df\": rpc error: code = NotFound desc = could not find container \"6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df\": container with ID starting with 6189a525afc38b4dec4013389441a89916d88a5c1740530910df9f5dd2d2b2df not found: ID does not exist"
Jul 08 16:58:47.657117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.657036    2559 scope.go:117] "RemoveContainer" containerID="21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea"
Jul 08 16:58:47.657390 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:58:47.657359    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea\": container with ID starting with 21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea not found: ID does not exist" containerID="21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea"
Jul 08 16:58:47.657390 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.657383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea"} err="failed to get container status \"21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea\": rpc error: code = NotFound desc = could not find container \"21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea\": container with ID starting with 21c64b848856896504fe36e29cbe6a2fe6be0362d1f649e792e04712123223ea not found: ID does not exist"
Jul 08 16:58:47.657539 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.657398    2559 scope.go:117] "RemoveContainer" containerID="bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084"
Jul 08 16:58:47.657657 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:58:47.657639    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084\": container with ID starting with bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084 not found: ID does not exist" containerID="bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084"
Jul 08 16:58:47.657937 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.657662    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084"} err="failed to get container status \"bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084\": rpc error: code = NotFound desc = could not find container \"bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084\": container with ID starting with bba3d65de74cfb4e2cb0f79b2c5147d38b3671151e9d04f0f2cb920a47162084 not found: ID does not exist"
Jul 08 16:58:47.658193 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.658177    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"]
Jul 08 16:58:47.660899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:47.660878    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zcsbb"]
Jul 08 16:58:48.672516 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:48.672486    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" path="/var/lib/kubelet/pods/989097cc-9869-4539-a82d-f7acf3d2da14/volumes"
Jul 08 16:58:50.642033 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:50.642005    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:58:50.646868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:50.646843    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 16:58:51.447644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.447610    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:51.530071 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530038    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"]
Jul 08 16:58:51.530373 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530357    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="storage-initializer"
Jul 08 16:58:51.530447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530376    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="storage-initializer"
Jul 08 16:58:51.530447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530399    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kserve-container"
Jul 08 16:58:51.530447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530408    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kserve-container"
Jul 08 16:58:51.530447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530429    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kube-rbac-proxy"
Jul 08 16:58:51.530447 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530437    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kube-rbac-proxy"
Jul 08 16:58:51.530736 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530508    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kube-rbac-proxy"
Jul 08 16:58:51.530736 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.530525    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="989097cc-9869-4539-a82d-f7acf3d2da14" containerName="kserve-container"
Jul 08 16:58:51.533503 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.533481    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.535426 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 08 16:58:51.535525 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535414    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:58:51.535525 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535511    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 08 16:58:51.535654 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535546    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:58:51.535877 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:58:51.535964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535875    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:58:51.535964 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.535888    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-924bj\""
Jul 08 16:58:51.551288 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.551267    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"]
Jul 08 16:58:51.585454 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.585424    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:51.645544 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.645519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c48f453-e7fe-4ff8-89e9-0ee800047686-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.645874 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.645566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.645874 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.645624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdvnp\" (UniqueName: \"kubernetes.io/projected/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-api-access-wdvnp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.645874 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.645651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.645874 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.645698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c48f453-e7fe-4ff8-89e9-0ee800047686-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747017 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.746996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c48f453-e7fe-4ff8-89e9-0ee800047686-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747149 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747149 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdvnp\" (UniqueName: \"kubernetes.io/projected/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-api-access-wdvnp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747149 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747149 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c48f453-e7fe-4ff8-89e9-0ee800047686-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747409 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747387    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c48f453-e7fe-4ff8-89e9-0ee800047686-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747746 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747725    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.747817 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.747745    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.749518 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.749498    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c48f453-e7fe-4ff8-89e9-0ee800047686-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.757259 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.757237    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdvnp\" (UniqueName: \"kubernetes.io/projected/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-api-access-wdvnp\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.843915 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.843888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:58:51.974387 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:51.974316    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"]
Jul 08 16:58:51.976614 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:58:51.976586    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c48f453_e7fe_4ff8_89e9_0ee800047686.slice/crio-33a4d39a2f26873962064a95f0192f83b7162457acdf5186b6083e56a996c6cb WatchSource:0}: Error finding container 33a4d39a2f26873962064a95f0192f83b7162457acdf5186b6083e56a996c6cb: Status 404 returned error can't find the container with id 33a4d39a2f26873962064a95f0192f83b7162457acdf5186b6083e56a996c6cb
Jul 08 16:58:52.650041 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:52.650011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerStarted","Data":"dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310"}
Jul 08 16:58:52.650450 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:58:52.650069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerStarted","Data":"33a4d39a2f26873962064a95f0192f83b7162457acdf5186b6083e56a996c6cb"}
Jul 08 16:59:01.691525 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:01.691487    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerID="dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310" exitCode=0
Jul 08 16:59:01.691940 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:01.691561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerDied","Data":"dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310"}
Jul 08 16:59:02.697536 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:02.697501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerStarted","Data":"ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3"}
Jul 08 16:59:02.698052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:02.697569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerStarted","Data":"2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f"}
Jul 08 16:59:02.698052 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:02.697782    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:59:02.719868 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:02.719820    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" podStartSLOduration=2.654308498 podStartE2EDuration="11.719806051s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.065497553s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:59:02.718716212 +0000 UTC m=+1512.608131222" watchObservedRunningTime="2026-07-08 16:59:02.719806051 +0000 UTC m=+1512.609221061"
Jul 08 16:59:03.701675 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:03.701645    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:59:03.703016 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:03.702986    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 08 16:59:04.705370 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:04.705330    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 08 16:59:09.710334 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:09.710303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:59:09.710822 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:09.710798    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 08 16:59:11.096899 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:11.096863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:19.711675 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:19.711602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:59:21.520590 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:21.520559    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"]
Jul 08 16:59:21.521123 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:21.520860    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container" containerID="cri-o://2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f" gracePeriod=30
Jul 08 16:59:21.521123 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:21.520895    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kube-rbac-proxy" containerID="cri-o://ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3" gracePeriod=30
Jul 08 16:59:21.547726 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:21.547696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:21.777173 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:21.777081    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerID="ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3" exitCode=2
Jul 08 16:59:21.777173 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:21.777117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerDied","Data":"ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3"}
Jul 08 16:59:22.357446 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.357424    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:59:22.488944 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.488915    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c48f453-e7fe-4ff8-89e9-0ee800047686-kserve-provision-location\") pod \"2c48f453-e7fe-4ff8-89e9-0ee800047686\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") "
Jul 08 16:59:22.489125 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.488965    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdvnp\" (UniqueName: \"kubernetes.io/projected/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-api-access-wdvnp\") pod \"2c48f453-e7fe-4ff8-89e9-0ee800047686\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") "
Jul 08 16:59:22.489125 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.488985    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-cabundle-cert\") pod \"2c48f453-e7fe-4ff8-89e9-0ee800047686\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") "
Jul 08 16:59:22.489125 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.489005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c48f453-e7fe-4ff8-89e9-0ee800047686-proxy-tls\") pod \"2c48f453-e7fe-4ff8-89e9-0ee800047686\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") "
Jul 08 16:59:22.489125 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.489063    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-rbac-proxy-sar-config\") pod \"2c48f453-e7fe-4ff8-89e9-0ee800047686\" (UID: \"2c48f453-e7fe-4ff8-89e9-0ee800047686\") "
Jul 08 16:59:22.489378 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.489351    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c48f453-e7fe-4ff8-89e9-0ee800047686" (UID: "2c48f453-e7fe-4ff8-89e9-0ee800047686"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:22.489469 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.489446    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c48f453-e7fe-4ff8-89e9-0ee800047686" (UID: "2c48f453-e7fe-4ff8-89e9-0ee800047686"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:22.489927 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.489907    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c48f453-e7fe-4ff8-89e9-0ee800047686-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c48f453-e7fe-4ff8-89e9-0ee800047686" (UID: "2c48f453-e7fe-4ff8-89e9-0ee800047686"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:22.491041 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.491019    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c48f453-e7fe-4ff8-89e9-0ee800047686-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c48f453-e7fe-4ff8-89e9-0ee800047686" (UID: "2c48f453-e7fe-4ff8-89e9-0ee800047686"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:59:22.491127 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.491106    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-api-access-wdvnp" (OuterVolumeSpecName: "kube-api-access-wdvnp") pod "2c48f453-e7fe-4ff8-89e9-0ee800047686" (UID: "2c48f453-e7fe-4ff8-89e9-0ee800047686"). InnerVolumeSpecName "kube-api-access-wdvnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:59:22.589798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.589767    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:59:22.589798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.589793    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c48f453-e7fe-4ff8-89e9-0ee800047686-kserve-provision-location\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:59:22.589798 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.589804    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdvnp\" (UniqueName: \"kubernetes.io/projected/2c48f453-e7fe-4ff8-89e9-0ee800047686-kube-api-access-wdvnp\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:59:22.590214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.589813    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c48f453-e7fe-4ff8-89e9-0ee800047686-cabundle-cert\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:59:22.590214 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.589822    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c48f453-e7fe-4ff8-89e9-0ee800047686-proxy-tls\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 16:59:22.782853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.782774    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerID="2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f" exitCode=0
Jul 08 16:59:22.782853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.782822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerDied","Data":"2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f"}
Jul 08 16:59:22.782853 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.782827    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"
Jul 08 16:59:22.783090 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.782869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt" event={"ID":"2c48f453-e7fe-4ff8-89e9-0ee800047686","Type":"ContainerDied","Data":"33a4d39a2f26873962064a95f0192f83b7162457acdf5186b6083e56a996c6cb"}
Jul 08 16:59:22.783090 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.782901    2559 scope.go:117] "RemoveContainer" containerID="ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3"
Jul 08 16:59:22.791117 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.791098    2559 scope.go:117] "RemoveContainer" containerID="2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f"
Jul 08 16:59:22.798174 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.798159    2559 scope.go:117] "RemoveContainer" containerID="dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310"
Jul 08 16:59:22.801610 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.801592    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"]
Jul 08 16:59:22.805172 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.805149    2559 scope.go:117] "RemoveContainer" containerID="ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3"
Jul 08 16:59:22.805412 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:59:22.805393    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3\": container with ID starting with ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3 not found: ID does not exist" containerID="ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3"
Jul 08 16:59:22.805470 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.805421    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3"} err="failed to get container status \"ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3\": rpc error: code = NotFound desc = could not find container \"ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3\": container with ID starting with ba470cfe8d940beaf6575526dcbd9b82ecfac125a31b2da0bb85645274dc02e3 not found: ID does not exist"
Jul 08 16:59:22.805470 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.805437    2559 scope.go:117] "RemoveContainer" containerID="2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f"
Jul 08 16:59:22.805674 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:59:22.805659    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f\": container with ID starting with 2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f not found: ID does not exist" containerID="2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f"
Jul 08 16:59:22.805739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.805692    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f"} err="failed to get container status \"2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f\": rpc error: code = NotFound desc = could not find container \"2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f\": container with ID starting with 2a7bdcbf006fc3a41159d2c6024a520fdd3b61d622049b05c17293add6a5eb4f not found: ID does not exist"
Jul 08 16:59:22.805739 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.805708    2559 scope.go:117] "RemoveContainer" containerID="dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310"
Jul 08 16:59:22.805930 ip-10-0-137-118 kubenswrapper[2559]: E0708 16:59:22.805910    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310\": container with ID starting with dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310 not found: ID does not exist" containerID="dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310"
Jul 08 16:59:22.805979 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.805935    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310"} err="failed to get container status \"dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310\": rpc error: code = NotFound desc = could not find container \"dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310\": container with ID starting with dab81d6f02b9a39943252934737411ce9103e800aaecb5f8e305447f77ceb310 not found: ID does not exist"
Jul 08 16:59:22.809045 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:22.809024    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-schzt"]
Jul 08 16:59:24.672859 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:24.672828    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" path="/var/lib/kubelet/pods/2c48f453-e7fe-4ff8-89e9-0ee800047686/volumes"
Jul 08 16:59:28.313210 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313180    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5gszk/must-gather-rv2vz"]
Jul 08 16:59:28.313656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313571    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container"
Jul 08 16:59:28.313656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313588    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container"
Jul 08 16:59:28.313656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313603    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="storage-initializer"
Jul 08 16:59:28.313656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313611    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="storage-initializer"
Jul 08 16:59:28.313656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313641    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kube-rbac-proxy"
Jul 08 16:59:28.313656 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313648    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kube-rbac-proxy"
Jul 08 16:59:28.313994 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313739    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kube-rbac-proxy"
Jul 08 16:59:28.313994 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.313752    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c48f453-e7fe-4ff8-89e9-0ee800047686" containerName="kserve-container"
Jul 08 16:59:28.318203 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.318181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.320577 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.320556    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5gszk\"/\"kube-root-ca.crt\""
Jul 08 16:59:28.320956 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.320942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5gszk\"/\"openshift-service-ca.crt\""
Jul 08 16:59:28.321012 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.320944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5gszk\"/\"default-dockercfg-2xdgb\""
Jul 08 16:59:28.325164 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.324847    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gszk/must-gather-rv2vz"]
Jul 08 16:59:28.428612 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.428586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s66f9\" (UniqueName: \"kubernetes.io/projected/0a354db4-7c57-4391-ba0f-1743aab712bb-kube-api-access-s66f9\") pod \"must-gather-rv2vz\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") " pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.428730 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.428630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a354db4-7c57-4391-ba0f-1743aab712bb-must-gather-output\") pod \"must-gather-rv2vz\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") " pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.529889 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.529866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a354db4-7c57-4391-ba0f-1743aab712bb-must-gather-output\") pod \"must-gather-rv2vz\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") " pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.529982 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.529917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66f9\" (UniqueName: \"kubernetes.io/projected/0a354db4-7c57-4391-ba0f-1743aab712bb-kube-api-access-s66f9\") pod \"must-gather-rv2vz\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") " pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.530181 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.530162    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a354db4-7c57-4391-ba0f-1743aab712bb-must-gather-output\") pod \"must-gather-rv2vz\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") " pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.538735 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.538711    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66f9\" (UniqueName: \"kubernetes.io/projected/0a354db4-7c57-4391-ba0f-1743aab712bb-kube-api-access-s66f9\") pod \"must-gather-rv2vz\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") " pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.627846 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.627784    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 16:59:28.753867 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.753837    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gszk/must-gather-rv2vz"]
Jul 08 16:59:28.756078 ip-10-0-137-118 kubenswrapper[2559]: W0708 16:59:28.756051    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a354db4_7c57_4391_ba0f_1743aab712bb.slice/crio-a3306c8891deb89470381e86eab2d240c1f66a90ccae001b9f19e72a6dc91c4e WatchSource:0}: Error finding container a3306c8891deb89470381e86eab2d240c1f66a90ccae001b9f19e72a6dc91c4e: Status 404 returned error can't find the container with id a3306c8891deb89470381e86eab2d240c1f66a90ccae001b9f19e72a6dc91c4e
Jul 08 16:59:28.757693 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.757661    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:59:28.807359 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:28.807330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gszk/must-gather-rv2vz" event={"ID":"0a354db4-7c57-4391-ba0f-1743aab712bb","Type":"ContainerStarted","Data":"a3306c8891deb89470381e86eab2d240c1f66a90ccae001b9f19e72a6dc91c4e"}
Jul 08 16:59:29.439358 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:29.439324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:29.515107 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:29.515069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:35.845851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:35.845809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gszk/must-gather-rv2vz" event={"ID":"0a354db4-7c57-4391-ba0f-1743aab712bb","Type":"ContainerStarted","Data":"b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55"}
Jul 08 16:59:35.845851 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:35.845855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gszk/must-gather-rv2vz" event={"ID":"0a354db4-7c57-4391-ba0f-1743aab712bb","Type":"ContainerStarted","Data":"531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"}
Jul 08 16:59:35.871176 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:35.871126    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5gszk/must-gather-rv2vz" podStartSLOduration=1.549737897 podStartE2EDuration="7.871112035s" totalImagesPullingTime="6.321374138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:59:35.869923577 +0000 UTC m=+1545.759338587" watchObservedRunningTime="2026-07-08 16:59:35.871112035 +0000 UTC m=+1545.760527045"
Jul 08 16:59:55.936534 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:55.936506    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerID="531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad" exitCode=0
Jul 08 16:59:55.936900 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:55.936583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gszk/must-gather-rv2vz" event={"ID":"0a354db4-7c57-4391-ba0f-1743aab712bb","Type":"ContainerDied","Data":"531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"}
Jul 08 16:59:55.936948 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:55.936905    2559 scope.go:117] "RemoveContainer" containerID="531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"
Jul 08 16:59:56.888612 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:56.888573    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5gszk_must-gather-rv2vz_0a354db4-7c57-4391-ba0f-1743aab712bb/gather/0.log"
Jul 08 16:59:57.275644 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:57.275612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:57.422108 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:57.422070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:57.522121 ip-10-0-137-118 kubenswrapper[2559]: I0708 16:59:57.522089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:00.433407 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:00.433374    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kn2hm_598f3885-6ab6-4276-a3f0-8b6e634cf091/global-pull-secret-syncer/0.log"
Jul 08 17:00:00.491862 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:00.491835    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log"
Jul 08 17:00:00.642387 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:00.642358    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-118.ec2.internal_e23d70a312ac711b759fd35bb366a792/haproxy/0.log"
Jul 08 17:00:02.255268 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.255237    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5gszk/must-gather-rv2vz"]
Jul 08 17:00:02.255664 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.255420    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-5gszk/must-gather-rv2vz" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="copy" containerID="cri-o://b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55" gracePeriod=2
Jul 08 17:00:02.261521 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.261495    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5gszk/must-gather-rv2vz"]
Jul 08 17:00:02.262488 ip-10-0-137-118 kubenswrapper[2559]: E0708 17:00:02.262459    2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0a354db4-7c57-4391-ba0f-1743aab712bb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-5gszk\"/\"must-gather-rv2vz\": pods \"must-gather-rv2vz\" not found" pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 17:00:02.484129 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.484095    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5gszk_must-gather-rv2vz_0a354db4-7c57-4391-ba0f-1743aab712bb/copy/0.log"
Jul 08 17:00:02.486660 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.486642    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 17:00:02.488073 ip-10-0-137-118 kubenswrapper[2559]: E0708 17:00:02.488049    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rv2vz\" is forbidden: User \"system:node:ip-10-0-137-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5gszk\": no relationship found between node 'ip-10-0-137-118.ec2.internal' and this object" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 17:00:02.602822 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.602764    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s66f9\" (UniqueName: \"kubernetes.io/projected/0a354db4-7c57-4391-ba0f-1743aab712bb-kube-api-access-s66f9\") pod \"0a354db4-7c57-4391-ba0f-1743aab712bb\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") "
Jul 08 17:00:02.602920 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.602841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a354db4-7c57-4391-ba0f-1743aab712bb-must-gather-output\") pod \"0a354db4-7c57-4391-ba0f-1743aab712bb\" (UID: \"0a354db4-7c57-4391-ba0f-1743aab712bb\") "
Jul 08 17:00:02.604104 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.604083    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a354db4-7c57-4391-ba0f-1743aab712bb-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0a354db4-7c57-4391-ba0f-1743aab712bb" (UID: "0a354db4-7c57-4391-ba0f-1743aab712bb"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:00:02.604847 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.604826    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a354db4-7c57-4391-ba0f-1743aab712bb-kube-api-access-s66f9" (OuterVolumeSpecName: "kube-api-access-s66f9") pod "0a354db4-7c57-4391-ba0f-1743aab712bb" (UID: "0a354db4-7c57-4391-ba0f-1743aab712bb"). InnerVolumeSpecName "kube-api-access-s66f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:00:02.672622 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.672585    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" path="/var/lib/kubelet/pods/0a354db4-7c57-4391-ba0f-1743aab712bb/volumes"
Jul 08 17:00:02.703718 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.703675    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s66f9\" (UniqueName: \"kubernetes.io/projected/0a354db4-7c57-4391-ba0f-1743aab712bb-kube-api-access-s66f9\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 17:00:02.703718 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.703717    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a354db4-7c57-4391-ba0f-1743aab712bb-must-gather-output\") on node \"ip-10-0-137-118.ec2.internal\" DevicePath \"\""
Jul 08 17:00:02.964199 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.964136    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5gszk_must-gather-rv2vz_0a354db4-7c57-4391-ba0f-1743aab712bb/copy/0.log"
Jul 08 17:00:02.964498 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.964475    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerID="b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55" exitCode=143
Jul 08 17:00:02.964549 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.964521    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gszk/must-gather-rv2vz"
Jul 08 17:00:02.964590 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.964565    2559 scope.go:117] "RemoveContainer" containerID="b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55"
Jul 08 17:00:02.971572 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.971559    2559 scope.go:117] "RemoveContainer" containerID="531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"
Jul 08 17:00:02.983335 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.983319    2559 scope.go:117] "RemoveContainer" containerID="b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55"
Jul 08 17:00:02.983593 ip-10-0-137-118 kubenswrapper[2559]: E0708 17:00:02.983574    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55\": container with ID starting with b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55 not found: ID does not exist" containerID="b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55"
Jul 08 17:00:02.983691 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.983601    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55"} err="failed to get container status \"b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55\": rpc error: code = NotFound desc = could not find container \"b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55\": container with ID starting with b091a06f9f1b636ca1821ce90afcd21d7abe56b8975f736b851625759e872c55 not found: ID does not exist"
Jul 08 17:00:02.983691 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.983618    2559 scope.go:117] "RemoveContainer" containerID="531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"
Jul 08 17:00:02.983855 ip-10-0-137-118 kubenswrapper[2559]: E0708 17:00:02.983838    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad\": container with ID starting with 531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad not found: ID does not exist" containerID="531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"
Jul 08 17:00:02.983895 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:02.983859    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad"} err="failed to get container status \"531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad\": rpc error: code = NotFound desc = could not find container \"531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad\": container with ID starting with 531255f4af13ca0e7dc96952621dba8bc9a460efaf7b09d14cfbef5ff70c46ad not found: ID does not exist"
Jul 08 17:00:03.434510 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:03.434480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:03.795216 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:03.795184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdrjd_90db286d-0405-4190-8aa9-42695ef44a41/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:05.066513 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.066423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-66xjf_3e8cf99a-08eb-48a4-810c-766d6066a86a/monitoring-plugin/0.log"
Jul 08 17:00:05.103943 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.103920    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffksz_8ef92402-a218-4be9-b401-f24ffbefb85c/node-exporter/0.log"
Jul 08 17:00:05.128019 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.127995    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffksz_8ef92402-a218-4be9-b401-f24ffbefb85c/kube-rbac-proxy/0.log"
Jul 08 17:00:05.160582 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.160562    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffksz_8ef92402-a218-4be9-b401-f24ffbefb85c/init-textfile/0.log"
Jul 08 17:00:05.504658 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.504629    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/prometheus/0.log"
Jul 08 17:00:05.530142 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.530114    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/config-reloader/0.log"
Jul 08 17:00:05.553910 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.553887    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/thanos-sidecar/0.log"
Jul 08 17:00:05.577856 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.577836    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/kube-rbac-proxy-web/0.log"
Jul 08 17:00:05.600151 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.600131    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/kube-rbac-proxy/0.log"
Jul 08 17:00:05.625442 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.625421    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/kube-rbac-proxy-thanos/0.log"
Jul 08 17:00:05.652396 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.652373    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e37608cc-9544-49fe-8697-a6a91e1282e7/init-config-reloader/0.log"
Jul 08 17:00:05.862796 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.862715    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f75d6b49-kbb8r_d5f69410-a14a-4919-bfb2-f7d54e7d50f2/thanos-query/0.log"
Jul 08 17:00:05.885280 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.885255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f75d6b49-kbb8r_d5f69410-a14a-4919-bfb2-f7d54e7d50f2/kube-rbac-proxy-web/0.log"
Jul 08 17:00:05.911909 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.911885    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f75d6b49-kbb8r_d5f69410-a14a-4919-bfb2-f7d54e7d50f2/kube-rbac-proxy/0.log"
Jul 08 17:00:05.935659 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.935637    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f75d6b49-kbb8r_d5f69410-a14a-4919-bfb2-f7d54e7d50f2/prom-label-proxy/0.log"
Jul 08 17:00:05.961633 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.961610    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f75d6b49-kbb8r_d5f69410-a14a-4919-bfb2-f7d54e7d50f2/kube-rbac-proxy-rules/0.log"
Jul 08 17:00:05.985851 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:05.985835    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f75d6b49-kbb8r_d5f69410-a14a-4919-bfb2-f7d54e7d50f2/kube-rbac-proxy-metrics/0.log"
Jul 08 17:00:09.086089 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.086059    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-5wjj5_da4a3ca2-873f-4659-80f1-829ae85b4edf/volume-data-source-validator/0.log"
Jul 08 17:00:09.236394 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.236360    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"]
Jul 08 17:00:09.237351 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.237320    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="copy"
Jul 08 17:00:09.237526 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.237515    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="copy"
Jul 08 17:00:09.237652 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.237642    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="gather"
Jul 08 17:00:09.237764 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.237754    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="gather"
Jul 08 17:00:09.238027 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.238014    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="copy"
Jul 08 17:00:09.238146 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.238137    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a354db4-7c57-4391-ba0f-1743aab712bb" containerName="gather"
Jul 08 17:00:09.243946 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.243924    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.245870 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.245851    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xjj5\"/\"kube-root-ca.crt\""
Jul 08 17:00:09.246240 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.246222    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xjj5\"/\"default-dockercfg-2zhv5\""
Jul 08 17:00:09.246284 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.246247    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xjj5\"/\"openshift-service-ca.crt\""
Jul 08 17:00:09.248375 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.248353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"]
Jul 08 17:00:09.351131 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.351068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-podres\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.351243 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.351131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-lib-modules\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.351243 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.351173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-proc\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.351243 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.351196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbrp\" (UniqueName: \"kubernetes.io/projected/35488608-029f-4410-a0cc-79fa633f1f4d-kube-api-access-2bbrp\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.351243 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.351237    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-sys\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.451772 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-podres\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.451884 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-lib-modules\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.451884 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-proc\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.451884 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbrp\" (UniqueName: \"kubernetes.io/projected/35488608-029f-4410-a0cc-79fa633f1f4d-kube-api-access-2bbrp\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.452006 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451915    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-podres\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.452006 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-sys\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.452006 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451946    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-proc\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.452006 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.451965    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-lib-modules\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.452164 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.452048    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35488608-029f-4410-a0cc-79fa633f1f4d-sys\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.464641 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.464616    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbrp\" (UniqueName: \"kubernetes.io/projected/35488608-029f-4410-a0cc-79fa633f1f4d-kube-api-access-2bbrp\") pod \"perf-node-gather-daemonset-mpwrw\" (UID: \"35488608-029f-4410-a0cc-79fa633f1f4d\") " pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.555012 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.554989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:09.678164 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.678140    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"]
Jul 08 17:00:09.680417 ip-10-0-137-118 kubenswrapper[2559]: W0708 17:00:09.680391    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35488608_029f_4410_a0cc_79fa633f1f4d.slice/crio-e54951d79ff3721b023257a27984e2fd4a56a42be375c9201416139257c873df WatchSource:0}: Error finding container e54951d79ff3721b023257a27984e2fd4a56a42be375c9201416139257c873df: Status 404 returned error can't find the container with id e54951d79ff3721b023257a27984e2fd4a56a42be375c9201416139257c873df
Jul 08 17:00:09.993162 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.993133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw" event={"ID":"35488608-029f-4410-a0cc-79fa633f1f4d","Type":"ContainerStarted","Data":"223301bbefc17a17b6fa697debde9c5810ae475d18a1d3fbd0d7141378c03faf"}
Jul 08 17:00:09.993316 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.993185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw" event={"ID":"35488608-029f-4410-a0cc-79fa633f1f4d","Type":"ContainerStarted","Data":"e54951d79ff3721b023257a27984e2fd4a56a42be375c9201416139257c873df"}
Jul 08 17:00:09.993316 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:09.993223    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:10.010358 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:10.010319    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw" podStartSLOduration=1.010302099 podStartE2EDuration="1.010302099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:00:10.009861963 +0000 UTC m=+1579.899276974" watchObservedRunningTime="2026-07-08 17:00:10.010302099 +0000 UTC m=+1579.899717088"
Jul 08 17:00:10.306466 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:10.306395    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hlm6j_b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca/dns/0.log"
Jul 08 17:00:10.334936 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:10.334912    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hlm6j_b9e8ff94-3fcc-4a12-adb9-90ebd08e67ca/kube-rbac-proxy/0.log"
Jul 08 17:00:10.491796 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:10.491769    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lhssb_946032cc-95d9-4a74-ab85-10e9c5bc9051/dns-node-resolver/0.log"
Jul 08 17:00:11.109110 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:11.109083    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sjfpx_d19257c2-bde8-4433-bfdc-71e91f1c3caa/node-ca/0.log"
Jul 08 17:00:12.530699 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:12.530656    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mz246_64fe723a-e250-4875-a8db-4a2186f92a51/serve-healthcheck-canary/0.log"
Jul 08 17:00:12.973817 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:12.973743    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h242d_73409722-b9d0-4343-8331-9bc7e60d5f2e/kube-rbac-proxy/0.log"
Jul 08 17:00:12.995946 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:12.995921    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h242d_73409722-b9d0-4343-8331-9bc7e60d5f2e/exporter/0.log"
Jul 08 17:00:13.019456 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:13.019432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h242d_73409722-b9d0-4343-8331-9bc7e60d5f2e/extractor/0.log"
Jul 08 17:00:15.484970 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:15.484941    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-f8gsq_52c54020-3aeb-47f0-9395-faebb4a13865/s3-init/0.log"
Jul 08 17:00:15.502287 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:15.502265    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-f8gsq_52c54020-3aeb-47f0-9395-faebb4a13865/download-hf-model/0.log"
Jul 08 17:00:15.536032 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:15.536008    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-pfttt_15f08b71-1d2c-4f82-a9c9-cb6058a4b250/s3-tls-init-custom/0.log"
Jul 08 17:00:15.555419 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:15.555402    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-pfttt_15f08b71-1d2c-4f82-a9c9-cb6058a4b250/download-hf-model/0.log"
Jul 08 17:00:15.589461 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:15.589435    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hlzbj_bd22161f-4aaa-40f7-b4db-6268b613a2af/s3-tls-init-serving/0.log"
Jul 08 17:00:15.610409 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:15.610390    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hlzbj_bd22161f-4aaa-40f7-b4db-6268b613a2af/download-hf-model/0.log"
Jul 08 17:00:16.008815 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:16.008790    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xjj5/perf-node-gather-daemonset-mpwrw"
Jul 08 17:00:22.393444 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.393414    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/kube-multus-additional-cni-plugins/0.log"
Jul 08 17:00:22.420400 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.420371    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/egress-router-binary-copy/0.log"
Jul 08 17:00:22.448153 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.448133    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/cni-plugins/0.log"
Jul 08 17:00:22.474456 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.474435    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/bond-cni-plugin/0.log"
Jul 08 17:00:22.498870 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.498855    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/routeoverride-cni/0.log"
Jul 08 17:00:22.521863 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.521828    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/whereabouts-cni-bincopy/0.log"
Jul 08 17:00:22.544490 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.544467    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b45pc_fbbf911d-083c-4b51-a7b4-b795abbc5e70/whereabouts-cni/0.log"
Jul 08 17:00:22.782118 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.782087    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gz6lt_dcfc731a-fc64-46f4-bdc7-ebb83b39d5aa/kube-multus/0.log"
Jul 08 17:00:22.981118 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:22.981097    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-txdt2_33c52f87-ae67-40cb-8711-1a37de967be6/network-metrics-daemon/0.log"
Jul 08 17:00:23.007765 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:23.007740    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-txdt2_33c52f87-ae67-40cb-8711-1a37de967be6/kube-rbac-proxy/0.log"
Jul 08 17:00:24.325973 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.325950    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-controller/0.log"
Jul 08 17:00:24.344238 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.344216    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/1.log"
Jul 08 17:00:24.344419 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.344398    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovn-acl-logging/0.log"
Jul 08 17:00:24.367080 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.367057    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/kube-rbac-proxy-node/0.log"
Jul 08 17:00:24.393824 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.393805    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 17:00:24.414812 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.414792    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/northd/0.log"
Jul 08 17:00:24.438284 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.438265    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/nbdb/0.log"
Jul 08 17:00:24.462358 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.462339    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/sbdb/0.log"
Jul 08 17:00:24.521958 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:24.521934    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xbnrp_6b63cc5a-ba78-4672-b37a-d49c28673f50/ovnkube-controller/0.log"
Jul 08 17:00:25.753356 ip-10-0-137-118 kubenswrapper[2559]: I0708 17:00:25.753327    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mlbdg_a8cffb6b-8aa7-482d-8ec2-e88558428be5/network-check-target-container/0.log"