Jul 08 16:33:27.935193 ip-10-0-138-233 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.541131    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.545859    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.545968    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546071    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546105    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546112    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546120    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546126    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:28.548573 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546132    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546138    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546142    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546147    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546152    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546157    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546162    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546175    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546183    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546191    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546199    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546206    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546211    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546217    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546222    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546227    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546232    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546237    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546242    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:28.550764 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546253    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546259    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546264    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546269    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546274    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546279    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546285    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546290    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546295    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546299    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546305    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546310    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546317    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546328    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546332    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546337    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546346    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546351    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546356    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546360    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:28.551636 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546365    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546370    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546375    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546381    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546386    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546396    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546401    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546407    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546411    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546416    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546420    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546425    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546430    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546435    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546440    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546445    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546449    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546454    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546467    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546472    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:28.552952 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546476    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546481    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546486    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546491    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546496    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546501    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546506    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546511    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546516    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546521    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546525    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546535    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546540    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546545    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546551    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546555    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546564    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546569    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546573    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546578    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:28.554222 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546583    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546587    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546599    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546604    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546609    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546616    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546622    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546626    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546631    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546636    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546641    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546646    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.546650    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548254    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548266    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548270    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548274    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548278    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548282    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:28.555539 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548288    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548293    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548297    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548301    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548304    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548308    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548311    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548315    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548318    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548322    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548325    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548329    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548332    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548336    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548339    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548343    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548346    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548349    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548353    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548356    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:28.557916 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548360    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548363    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548366    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548371    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548374    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548378    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548384    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548390    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548393    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548397    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548403    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548407    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548412    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548415    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548419    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548422    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548426    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548429    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548433    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548436    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:28.558479 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548440    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548443    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548446    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548450    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548456    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548460    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548463    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548467    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548471    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548474    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548477    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548481    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548484    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548488    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548491    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548495    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548499    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548502    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548508    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:28.559030 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548513    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548517    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548521    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548524    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548527    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548532    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548536    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548539    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548541    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548544    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548547    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548550    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548553    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548555    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548559    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548562    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548565    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548568    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548573    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548576    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:28.559522 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548579    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548581    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548584    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548588    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548590    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548593    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548596    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548599    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548601    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548604    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548607    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548614    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548619    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.548622    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549070    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549077    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549082    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549085    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549088    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:28.560046 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549091    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549094    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549097    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549099    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549103    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549105    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549108    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549111    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549113    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549116    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549119    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549122    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549125    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549128    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549131    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549133    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549136    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549139    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549141    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549144    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:28.560540 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549147    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549149    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549152    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549155    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549159    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549164    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549166    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549169    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549172    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549176    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549180    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549184    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549187    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549190    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549193    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549195    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549199    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549202    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549205    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:28.561082 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549208    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549210    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549213    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549216    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549218    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549225    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549228    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549231    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549234    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549236    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549239    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549242    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549244    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549247    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549250    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549253    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549256    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549260    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549264    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549267    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:28.561565 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549270    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549272    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549275    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549278    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549280    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549283    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549285    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549289    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549291    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549294    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549297    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549299    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549302    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549305    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549309    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549312    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549315    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549318    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549321    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549325    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:28.562101 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549328    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549331    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549333    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549336    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549339    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549342    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549344    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549347    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549350    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549354    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549358    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549361    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549364    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549366    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.549369    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549451    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549458    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549467    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549474    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549481    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549485    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:33:28.562638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549490    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549494    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549498    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549501    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549504    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549508    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549511    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549515    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549518    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549521    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549525    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549528    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549532    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549535    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549538    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549542    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549545    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549549    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549553    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549556    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549562    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549566    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549569    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549572    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:33:28.563200 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549575    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549579    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549583    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549586    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549589    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549592    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549595    2561 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549599    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549603    2561 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549606    2561 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549610    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549613    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549616    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549620    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549623    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549626    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549629    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549632    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549635    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549639    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549642    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549645    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549648    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549651    2561 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549655    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:33:28.563830 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549658    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549661    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549665    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549669    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549674    2561 flags.go:64] FLAG: --help="false"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549677    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549680    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549684    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549687    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549690    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549694    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549697    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549699    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549702    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549705    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549709    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549713    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549716    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549719    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549722    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549725    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549727    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549730    2561 flags.go:64] FLAG: --lock-file=""
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549733    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:33:28.564540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549736    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549740    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549745    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549749    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549752    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549756    2561 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549778    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549782    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549785    2561 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549788    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549794    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549798    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549804    2561 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549807    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549811    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549814    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549817    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549820    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549823    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549826    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549835    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549839    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549842    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:33:28.565190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549845    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549848    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549854    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549858    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549864    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549868    2561 flags.go:64] FLAG: --port="10250"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549872    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549876    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-029642b4fa8bb8064"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549879    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549882    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549885    2561 flags.go:64] FLAG: --register-node="true"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549888    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549892    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549897    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549900    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549903    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549907    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549910    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549914    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549916    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549921    2561 flags.go:64] FLAG: --runonce="false"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549924    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549929    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549932    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549936    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:33:28.565805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549938    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549942    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549945    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549948    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549951    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549954    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549957    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549960    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549963    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549967    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549970    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549976    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549979    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549982    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549986    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549989    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549992    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549995    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.549998    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.550001    2561 flags.go:64] FLAG: --v="2"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.550005    2561 flags.go:64] FLAG: --version="false"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.550010    2561 flags.go:64] FLAG: --vmodule=""
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.550019    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.550022    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:33:28.566553 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550143    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550147    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550150    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550154    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550158    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550162    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550165    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550168    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550171    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550174    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550176    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550179    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550182    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550185    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550187    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550190    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550192    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550195    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550198    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550201    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:28.567203 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550203    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550206    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550209    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550211    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550214    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550217    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550219    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550222    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550225    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550228    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550231    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550234    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550236    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550239    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550242    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550246    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550249    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550254    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550257    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550260    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:28.568143 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550263    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550265    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550268    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550271    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550274    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550277    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550279    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550282    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550286    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550291    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550294    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550297    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550300    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550303    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550305    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550308    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550311    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550314    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550317    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550319    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:28.568730 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550322    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550324    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550327    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550331    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550333    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550352    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550355    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550361    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550365    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550370    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550374    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550377    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550379    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550382    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550385    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550388    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550391    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550393    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550396    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550399    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:28.569278 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550402    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550404    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550407    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550410    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550413    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550417    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550420    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550424    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550427    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550430    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550433    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550436    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550440    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550443    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550447    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550450    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550453    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550457    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:28.569802 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.550459    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.551457    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.559571    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.559594    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.559621    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.559636    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559718    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559725    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559731    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559736    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559741    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559747    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559751    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559771    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559776    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:28.570277 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559781    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559786    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559790    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559794    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559799    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559803    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559808    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559812    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559817    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559822    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559826    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559830    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559835    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559839    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559843    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559850    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559856    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559860    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559864    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559869    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:28.570692 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559874    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559879    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559883    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559887    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559892    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559896    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559904    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559911    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559915    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559921    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559925    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559931    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559937    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559941    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559946    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559951    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559956    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559960    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559965    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:28.571249 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559970    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559974    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559978    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559982    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559989    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.559995    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560000    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560005    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560010    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560034    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560039    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560044    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560049    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560055    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560060    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560066    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560070    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560075    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560082    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560086    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:28.571742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560090    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560095    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560101    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560107    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560111    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560116    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560121    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560125    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560130    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560134    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560139    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560143    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560148    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560153    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560157    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560161    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560165    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560169    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560173    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560178    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:28.572288 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560190    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560194    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560199    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560204    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560208    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560214    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560219    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560224    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560228    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560232    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560236    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.560244    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560423    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560431    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560435    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:33:28.572835 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560440    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560445    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560449    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560453    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560458    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560465    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560472    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560477    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560481    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560486    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560490    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560495    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560499    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560504    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560509    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560513    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560517    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560524    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560529    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:33:28.573245 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560533    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560543    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560548    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560553    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560558    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560562    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560567    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560572    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560576    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560581    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560585    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560589    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560593    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560598    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560603    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560607    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560612    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560616    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560620    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560625    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:33:28.573750 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560629    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560633    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560638    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560642    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560646    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560650    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560655    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560660    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560665    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560670    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560674    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560678    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560682    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560687    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560692    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560696    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560701    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560706    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560710    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560715    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:33:28.574320 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560720    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560724    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560728    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560732    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560736    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560741    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560745    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560749    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560753    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560772    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560777    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560781    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560786    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560791    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560796    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560800    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560804    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560809    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560813    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:33:28.574842 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560818    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560825    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560831    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560836    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560841    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560846    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560852    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560857    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560867    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560872    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560876    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560881    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560886    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560890    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560895    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560900    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560904    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:33:28.575351 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:28.560909    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.560918    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.561085    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.563996    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.565708    2561 server.go:1019] "Starting client certificate rotation"
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.565793    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.565835    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.568612    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:33:28.575827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.570646    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:33:28.602636 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.602616    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:33:28.607465 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.607441    2561 log.go:25] "Validated CRI v1 image API"
Jul 08 16:33:28.609217 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.609194    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:33:28.611831 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.611807    2561 fs.go:135] Filesystem UUIDs: map[314d16c7-e350-4160-b728-e97376bf0637:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a2f5a017-61aa-48a2-9d85-30264b0481c8:/dev/nvme0n1p4]
Jul 08 16:33:28.611923 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.611829    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:33:28.618226 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.618101    2561 manager.go:216] Machine: {Timestamp:2026-07-08 16:33:28.616166481 +0000 UTC m=+0.554767744 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3060411 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2398c39d1d6ac6a044afa06ee3cad0 SystemUUID:ec2398c3-9d1d-6ac6-a044-afa06ee3cad0 BootID:e8138087-9953-487f-858a-db48c158aedd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:62:2f:55:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:62:2f:55:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:72:e1:fa:9b:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:33:28.618226 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.618221    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:33:28.618359 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.618311    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:33:28.619298 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.619265    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:33:28.619479 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.619301    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:33:28.620031 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.620019    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:33:28.620070 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.620034    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:33:28.620070 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.620043    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:33:28.620628 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.620615    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:33:28.620785 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.620769    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:33:28.621872 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.621860    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:33:28.625929 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.625915    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:33:28.626051 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626040    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:33:28.626094 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626075    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:33:28.626136 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626096    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:33:28.626136 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626110    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:33:28.626136 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626121    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:33:28.626136 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626132    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:33:28.626536 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626521    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:33:28.626580 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.626555    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:33:28.628537 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.628520    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:33:28.630609 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.630592    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:33:28.630684 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.630625    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:33:28.632679 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632664    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632684    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632691    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632699    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632710    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632716    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632726    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632735    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:33:28.632747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632747    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:33:28.632993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632766    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:33:28.632993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632806    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:33:28.632993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.632816    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:33:28.633573 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.633561    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:33:28.633573 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.633572    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:33:28.637026 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.637009    2561 server.go:1268] "Started kubelet"
Jul 08 16:33:28.637131 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.637101    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:33:28.638010 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.637605    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:33:28.638010 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.637836    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:33:28.638026 ip-10-0-138-233 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:33:28.639175 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.639108    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:33:28.639175 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.639118    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:33:28.639175 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.639150    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:33:28.639904 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.639884    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:33:28.640460 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.640446    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:33:28.643546 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.643521    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:33:28.646810 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.646737    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2fh8x"
Jul 08 16:33:28.647794 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.647775    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:33:28.647870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.647776    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:33:28.648127 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.648045    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:28.648455 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648438    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:33:28.648562 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648551    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:33:28.648681 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648439    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:33:28.648805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648789    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:33:28.648880 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648806    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:33:28.648880 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648859    2561 factory.go:55] Registering systemd factory
Jul 08 16:33:28.648880 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.648875    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:33:28.649204 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.649186    2561 factory.go:153] Registering CRI-O factory
Jul 08 16:33:28.649204 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.649206    2561 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:33:28.649315 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.649272    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:33:28.649315 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.649297    2561 factory.go:103] Registering Raw factory
Jul 08 16:33:28.649315 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.649313    2561 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:33:28.649784 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.649769    2561 manager.go:349] Starting recovery of all containers
Jul 08 16:33:28.652078 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.652049    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:33:28.652171 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.652146    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:33:28.653101 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.652041    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-233.ec2.internal.18c05da35b9ff31a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-233.ec2.internal,UID:ip-10-0-138-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-233.ec2.internal,},FirstTimestamp:2026-07-08 16:33:28.636977946 +0000 UTC m=+0.575579210,LastTimestamp:2026-07-08 16:33:28.636977946 +0000 UTC m=+0.575579210,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-233.ec2.internal,}"
Jul 08 16:33:28.656929 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.656816    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2fh8x"
Jul 08 16:33:28.660291 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.660272    2561 manager.go:354] Recovery completed
Jul 08 16:33:28.664565 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.664551    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.667111 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667095    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.667168 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667127    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.667168 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667138    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.667630 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667617    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:33:28.667630 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667628    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:33:28.667776 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667643    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:33:28.667776 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.667726    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.668918 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.668901    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.669016 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.668936    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.669016 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.668953    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.669644 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.669563    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-233.ec2.internal.18c05da35d6bc54e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-233.ec2.internal,UID:ip-10-0-138-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-233.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-233.ec2.internal,},FirstTimestamp:2026-07-08 16:33:28.667112782 +0000 UTC m=+0.605714046,LastTimestamp:2026-07-08 16:33:28.667112782 +0000 UTC m=+0.605714046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-233.ec2.internal,}"
Jul 08 16:33:28.670042 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.670027    2561 policy_none.go:49] "None policy: Start"
Jul 08 16:33:28.670096 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.670045    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:33:28.670096 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.670057    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:33:28.672268 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.672254    2561 policy_none.go:47] "Start"
Jul 08 16:33:28.712388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.712370    2561 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:33:28.712521 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.712409    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:33:28.712521 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.712420    2561 server.go:87] "Starting device plugin registration server"
Jul 08 16:33:28.712732 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.712714    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:33:28.712932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.712750    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:33:28.713058 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.713021    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:33:28.713123 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.713115    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:33:28.713181 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.713126    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:33:28.713514 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.713486    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:33:28.713603 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.713549    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:28.794536 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.794498    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:33:28.796697 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.795778    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:33:28.796697 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.795815    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:33:28.796697 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.795839    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:33:28.796697 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.795876    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:33:28.798734 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.798709    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:28.813416 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.813401    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.815717 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.815694    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.815824 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.815739    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.815824 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.815773    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.815824 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.815807    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.827085 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.827068    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.827148 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.827094    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-233.ec2.internal\": node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:28.844839 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.844814    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:28.896944 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.896907    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"]
Jul 08 16:33:28.897045 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.896992    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.898041 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.898021    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.898128 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.898056    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.898128 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.898070    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.900417 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.900403    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.900550 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.900532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.900618 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.900566    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.901238 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.901220    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.901238 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.901229    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.901388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.901250    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.901388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.901259    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.901388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.901279    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.901388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.901262    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.903574 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.903558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.903646 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.903586    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:33:28.904843 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.904828    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:33:28.904916 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.904856    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:33:28.904916 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.904874    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:33:28.932807 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.932757    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-233.ec2.internal\" not found" node="ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.937460 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.937440    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-233.ec2.internal\" not found" node="ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.945625 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:28.945601    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:28.950596 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.950547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a9307618ad481c92addb0219af76bb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal\" (UID: \"9a9307618ad481c92addb0219af76bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.950596 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.950574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a9307618ad481c92addb0219af76bb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal\" (UID: \"9a9307618ad481c92addb0219af76bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:28.950596 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:28.950594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/938243ce709c303fec6b51c38490eae8-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.ec2.internal\" (UID: \"938243ce709c303fec6b51c38490eae8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.046777 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.046743    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.051445 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.051422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/938243ce709c303fec6b51c38490eae8-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.ec2.internal\" (UID: \"938243ce709c303fec6b51c38490eae8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.051514 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.051456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a9307618ad481c92addb0219af76bb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal\" (UID: \"9a9307618ad481c92addb0219af76bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.051514 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.051475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a9307618ad481c92addb0219af76bb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal\" (UID: \"9a9307618ad481c92addb0219af76bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.051587 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.051538    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a9307618ad481c92addb0219af76bb7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal\" (UID: \"9a9307618ad481c92addb0219af76bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.051587 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.051568    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9a9307618ad481c92addb0219af76bb7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal\" (UID: \"9a9307618ad481c92addb0219af76bb7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.051587 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.051538    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/938243ce709c303fec6b51c38490eae8-config\") pod \"kube-apiserver-proxy-ip-10-0-138-233.ec2.internal\" (UID: \"938243ce709c303fec6b51c38490eae8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.147777 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.147737    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.240701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.240586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.240701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.240642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.248242 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.248220    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.353567 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.353523    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.454657 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.454619    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.555595 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.555498    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.565466 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.565438    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:33:29.565624 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.565605    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:33:29.579593 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.579561    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:29.656234 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:29.656199    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-233.ec2.internal\" not found"
Jul 08 16:33:29.663328 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.663302    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:28:29 +0000 UTC" deadline="2028-02-17 12:42:32.940341803 +0000 UTC"
Jul 08 16:33:29.663393 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.663328    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14132h9m3.277017002s"
Jul 08 16:33:29.663688 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.663669    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:29.748342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.748290    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.764119 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.764099    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:33:29.766193 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.766176    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal"
Jul 08 16:33:29.776213 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.776187    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:33:29.888035 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:29.888002    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9307618ad481c92addb0219af76bb7.slice/crio-2e64806f9ecd74e2c2b7c8bb261d42660d0e27a2ec8822613015eb9a8cb75ae6 WatchSource:0}: Error finding container 2e64806f9ecd74e2c2b7c8bb261d42660d0e27a2ec8822613015eb9a8cb75ae6: Status 404 returned error can't find the container with id 2e64806f9ecd74e2c2b7c8bb261d42660d0e27a2ec8822613015eb9a8cb75ae6
Jul 08 16:33:29.893376 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:29.893359    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:33:29.898395 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:29.898371    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938243ce709c303fec6b51c38490eae8.slice/crio-8a1b751f86685f0ac3668726c1ce2c689487d639204fbbeb4a9220e36fe6ee62 WatchSource:0}: Error finding container 8a1b751f86685f0ac3668726c1ce2c689487d639204fbbeb4a9220e36fe6ee62: Status 404 returned error can't find the container with id 8a1b751f86685f0ac3668726c1ce2c689487d639204fbbeb4a9220e36fe6ee62
Jul 08 16:33:30.099876 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.099851    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:30.627851 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.627806    2561 apiserver.go:52] "Watching apiserver"
Jul 08 16:33:30.638939 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.638906    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:33:30.640664 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.640633    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-xnrtk","openshift-network-operator/iptables-alerter-lnd8n","openshift-ovn-kubernetes/ovnkube-node-6m2cv","kube-system/global-pull-secret-syncer-dlb5z","kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bgpfl","openshift-dns/node-resolver-dcpxh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal","openshift-multus/multus-f6m68","openshift-multus/network-metrics-daemon-pdp4m","openshift-network-diagnostics/network-check-target-ddnsp","kube-system/konnectivity-agent-8lfxt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6","openshift-image-registry/node-ca-j8zcx"]
Jul 08 16:33:30.643443 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.643417    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.647351 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.646545    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.647351 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.646893    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.647351 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.647289    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q879r\""
Jul 08 16:33:30.650239 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.649544    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:33:30.651217 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.650955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:30.651217 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.651029    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:30.654839 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.654498    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.657370 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.657262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.657370 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.657331    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:30.658472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.658449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:33:30.658667 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.658650    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:33:30.659063 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.659041    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:33:30.659174 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.659068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vs8hx\""
Jul 08 16:33:30.659258 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.659241    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.659397 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.659379    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.659650 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.659636    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:33:30.661185 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.660789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.663665 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.663645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ksv6h\""
Jul 08 16:33:30.663903 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.663883    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.663991 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.663904    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.664170 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.664151    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:33:30.664314 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.664293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:30.664383 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.664331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/682cd642-9143-47ab-92ed-d33cfbd3eec9-hosts-file\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.664383 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.664359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682cd642-9143-47ab-92ed-d33cfbd3eec9-tmp-dir\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.664486 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.664387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4cj\" (UniqueName: \"kubernetes.io/projected/682cd642-9143-47ab-92ed-d33cfbd3eec9-kube-api-access-nv4cj\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.665821 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.665796    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:33:30.666227 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.666210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.666596 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.666578    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:33:30.668942 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.668313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.668942 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.668865    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbv4m\""
Jul 08 16:33:30.669513 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.669491    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.669632 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.669586    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.670921 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.670872    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:33:30.671461 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.671439    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.671905 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.671886    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z74s8\""
Jul 08 16:33:30.672153 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.672135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:33:30.672360 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.672342    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.673099 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.673079    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.675597 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.675576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.676602 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.676577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.676944 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.676924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q9dq2\""
Jul 08 16:33:30.677215 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.677198    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.677510 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.677487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:33:30.678294 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.678270    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:33:30.678709 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.678688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:30.678920 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.678786    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:30.678920 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.678887    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7pjxq\""
Jul 08 16:33:30.679088 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.679035    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:33:30.679182 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.679139    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:33:30.681259 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.681014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.683741 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.683401    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.683741 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.683594    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:33:30.684189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.684169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:33:30.684271 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.684209    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjzlh\""
Jul 08 16:33:30.687303 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.686046    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:33:30.687303 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.687223    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j682c"
Jul 08 16:33:30.687987 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.687966    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4mw5n\""
Jul 08 16:33:30.688364 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.688345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:33:30.697591 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.697566    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j682c"
Jul 08 16:33:30.750097 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.750068    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:33:30.764920 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.764888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysctl-d\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.765084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.764936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b19d6066-979b-4e85-9e24-6b20dc7ad427-konnectivity-ca\") pod \"konnectivity-agent-8lfxt\" (UID: \"b19d6066-979b-4e85-9e24-6b20dc7ad427\") " pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.765084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.764970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-systemd\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.765084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-log-socket\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.765084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqv7w\" (UniqueName: \"kubernetes.io/projected/b11058bb-8eed-4973-b455-a320ededc11b-kube-api-access-fqv7w\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:30.765084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-system-cni-dir\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-cni-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-ovn\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-host\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbqf\" (UniqueName: \"kubernetes.io/projected/d3c92389-4b78-4df8-88c7-be27ef0ce74e-kube-api-access-rwbqf\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4cj\" (UniqueName: \"kubernetes.io/projected/682cd642-9143-47ab-92ed-d33cfbd3eec9-kube-api-access-nv4cj\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvpb\" (UniqueName: \"kubernetes.io/projected/22afd025-a356-4e25-a21b-b1d8cd06dc7c-kube-api-access-jfvpb\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03e71b89-45d3-415c-844d-c81aa15d87f7-dbus\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.765357 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysctl-conf\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-env-overrides\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22afd025-a356-4e25-a21b-b1d8cd06dc7c-serviceca\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-kubelet\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-system-cni-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-modprobe-d\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-run\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.765725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-cnibin\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-k8s-cni-cncf-io\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22afd025-a356-4e25-a21b-b1d8cd06dc7c-host\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-sys-fs\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/682cd642-9143-47ab-92ed-d33cfbd3eec9-hosts-file\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-var-lib-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-node-log\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-cnibin\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.765965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9k5\" (UniqueName: \"kubernetes.io/projected/8d5696ce-42fc-4253-b113-85471cef9d7f-kube-api-access-pg9k5\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-os-release\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-netns\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766055    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/682cd642-9143-47ab-92ed-d33cfbd3eec9-hosts-file\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef661a59-4e98-4d7b-8a57-02b0ec567c42-host-slash\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysconfig\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-lib-modules\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-socket-dir-parent\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.766197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-cni-multus\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-conf-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8qm\" (UniqueName: \"kubernetes.io/projected/075bcaa7-4039-43d5-8c2a-a654180769f9-kube-api-access-xk8qm\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-socket-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdhp\" (UniqueName: \"kubernetes.io/projected/ef661a59-4e98-4d7b-8a57-02b0ec567c42-kube-api-access-pxdhp\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-os-release\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03e71b89-45d3-415c-844d-c81aa15d87f7-kubelet-config\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-sys\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/075bcaa7-4039-43d5-8c2a-a654180769f9-cni-binary-copy\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrnv\" (UniqueName: \"kubernetes.io/projected/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-kube-api-access-pfrnv\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-kubelet\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-run-netns\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-kubernetes\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-etc-kubernetes\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-systemd-units\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-etc-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6llk\" (UniqueName: \"kubernetes.io/projected/9cc40af6-5278-491d-83d0-7c61cc70943c-kube-api-access-p6llk\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-var-lib-kubelet\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-tuned\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-hostroot\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-multus-certs\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-cni-netd\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef661a59-4e98-4d7b-8a57-02b0ec567c42-iptables-alerter-script\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.766984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5696ce-42fc-4253-b113-85471cef9d7f-tmp\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-cni-bin\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-daemon-config\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b19d6066-979b-4e85-9e24-6b20dc7ad427-agent-certs\") pod \"konnectivity-agent-8lfxt\" (UID: \"b19d6066-979b-4e85-9e24-6b20dc7ad427\") " pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682cd642-9143-47ab-92ed-d33cfbd3eec9-tmp-dir\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.767949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovnkube-config\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-registration-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-device-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovnkube-script-lib\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-slash\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-systemd\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767314    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-cni-bin\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovn-node-metrics-cert\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.768773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.767539    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682cd642-9143-47ab-92ed-d33cfbd3eec9-tmp-dir\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.771461 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.771429    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:33:30.771461 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.771445    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:30.771666 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.771469    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:30.771666 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.771483    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:30.771666 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.771557    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:31.271535048 +0000 UTC m=+3.210136301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:30.776076 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.776047    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4cj\" (UniqueName: \"kubernetes.io/projected/682cd642-9143-47ab-92ed-d33cfbd3eec9-kube-api-access-nv4cj\") pod \"node-resolver-dcpxh\" (UID: \"682cd642-9143-47ab-92ed-d33cfbd3eec9\") " pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.802171 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.802111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal" event={"ID":"938243ce709c303fec6b51c38490eae8","Type":"ContainerStarted","Data":"8a1b751f86685f0ac3668726c1ce2c689487d639204fbbeb4a9220e36fe6ee62"}
Jul 08 16:33:30.804158 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.804120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal" event={"ID":"9a9307618ad481c92addb0219af76bb7","Type":"ContainerStarted","Data":"2e64806f9ecd74e2c2b7c8bb261d42660d0e27a2ec8822613015eb9a8cb75ae6"}
Jul 08 16:33:30.867657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-cni-bin\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.867657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-daemon-config\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b19d6066-979b-4e85-9e24-6b20dc7ad427-agent-certs\") pod \"konnectivity-agent-8lfxt\" (UID: \"b19d6066-979b-4e85-9e24-6b20dc7ad427\") " pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovnkube-config\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-registration-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-device-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovnkube-script-lib\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-slash\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.867949 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-systemd\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867960    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-cni-bin\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.867984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovn-node-metrics-cert\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysctl-d\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b19d6066-979b-4e85-9e24-6b20dc7ad427-konnectivity-ca\") pod \"konnectivity-agent-8lfxt\" (UID: \"b19d6066-979b-4e85-9e24-6b20dc7ad427\") " pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-systemd\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-log-socket\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqv7w\" (UniqueName: \"kubernetes.io/projected/b11058bb-8eed-4973-b455-a320ededc11b-kube-api-access-fqv7w\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-system-cni-dir\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-cni-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-ovn\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-host\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbqf\" (UniqueName: \"kubernetes.io/projected/d3c92389-4b78-4df8-88c7-be27ef0ce74e-kube-api-access-rwbqf\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvpb\" (UniqueName: \"kubernetes.io/projected/22afd025-a356-4e25-a21b-b1d8cd06dc7c-kube-api-access-jfvpb\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03e71b89-45d3-415c-844d-c81aa15d87f7-dbus\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.868459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysctl-conf\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-env-overrides\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22afd025-a356-4e25-a21b-b1d8cd06dc7c-serviceca\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-kubelet\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-system-cni-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868612    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovnkube-config\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.868708    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.868796    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:33:31.368757676 +0000 UTC m=+3.307358971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868853    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-modprobe-d\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868953    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-etc-selinux\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869105    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-daemon-config\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869172    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-cni-bin\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869378    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-registration-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.869395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869390    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-cni-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869437    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-device-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869452    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869503    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-slash\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869498    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-log-socket\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869504    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysctl-d\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-system-cni-dir\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869553    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-cni-bin\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869605    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869661    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-kubelet\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869804    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-systemd\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869861    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-systemd\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.868707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-modprobe-d\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869861    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-ovn\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-run\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-cnibin\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-k8s-cni-cncf-io\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.869985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22afd025-a356-4e25-a21b-b1d8cd06dc7c-host\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.870281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870001    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/22afd025-a356-4e25-a21b-b1d8cd06dc7c-serviceca\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-sys-fs\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870027    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-env-overrides\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-var-lib-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870071    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-sys-fs\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-node-log\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870108    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-host\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870133    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-system-cni-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-cnibin\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870155    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b19d6066-979b-4e85-9e24-6b20dc7ad427-konnectivity-ca\") pod \"konnectivity-agent-8lfxt\" (UID: \"b19d6066-979b-4e85-9e24-6b20dc7ad427\") " pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870165    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-cnibin\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870173    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870177    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-k8s-cni-cncf-io\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9k5\" (UniqueName: \"kubernetes.io/projected/8d5696ce-42fc-4253-b113-85471cef9d7f-kube-api-access-pg9k5\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870209    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22afd025-a356-4e25-a21b-b1d8cd06dc7c-host\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870232    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-var-lib-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870239    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.871176 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870241    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-run\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-os-release\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-netns\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870321    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysctl-conf\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef661a59-4e98-4d7b-8a57-02b0ec567c42-host-slash\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870362    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870368    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870387    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-netns\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870408    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef661a59-4e98-4d7b-8a57-02b0ec567c42-host-slash\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870418    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03e71b89-45d3-415c-844d-c81aa15d87f7-dbus\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysconfig\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870449    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-cnibin\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870449    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-os-release\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870461    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-node-log\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870466    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovnkube-script-lib\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-lib-modules\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-socket-dir-parent\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870555    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-sysconfig\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.872091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-cni-multus\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870614    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-lib-modules\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870631    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-var-lib-cni-multus\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870649    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-socket-dir-parent\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-conf-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8qm\" (UniqueName: \"kubernetes.io/projected/075bcaa7-4039-43d5-8c2a-a654180769f9-kube-api-access-xk8qm\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.870725    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-multus-conf-dir\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-socket-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdhp\" (UniqueName: \"kubernetes.io/projected/ef661a59-4e98-4d7b-8a57-02b0ec567c42-kube-api-access-pxdhp\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-os-release\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03e71b89-45d3-415c-844d-c81aa15d87f7-kubelet-config\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-sys\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/075bcaa7-4039-43d5-8c2a-a654180769f9-cni-binary-copy\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrnv\" (UniqueName: \"kubernetes.io/projected/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-kube-api-access-pfrnv\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-kubelet\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-run-netns\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.873073 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-kubernetes\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-etc-kubernetes\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-systemd-units\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-etc-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6llk\" (UniqueName: \"kubernetes.io/projected/9cc40af6-5278-491d-83d0-7c61cc70943c-kube-api-access-p6llk\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-var-lib-kubelet\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-tuned\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-hostroot\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-multus-certs\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871786    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-run-netns\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871843    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-cni-netd\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871962    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3c92389-4b78-4df8-88c7-be27ef0ce74e-socket-dir\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872219    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cc40af6-5278-491d-83d0-7c61cc70943c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872231    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cc40af6-5278-491d-83d0-7c61cc70943c-os-release\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872289    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03e71b89-45d3-415c-844d-c81aa15d87f7-kubelet-config\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:30.873961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872294    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-etc-kubernetes\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872337    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-kubernetes\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872343    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-systemd-units\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872381    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-sys\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872388    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-etc-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872612    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-kubelet\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.871794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-host-cni-netd\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef661a59-4e98-4d7b-8a57-02b0ec567c42-iptables-alerter-script\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5696ce-42fc-4253-b113-85471cef9d7f-tmp\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872856    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/075bcaa7-4039-43d5-8c2a-a654180769f9-cni-binary-copy\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.872870    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872921    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-ovn-node-metrics-cert\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872935    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-hostroot\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:30.872941    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:31.372918308 +0000 UTC m=+3.311519572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.872983    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/075bcaa7-4039-43d5-8c2a-a654180769f9-host-run-multus-certs\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.873032    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-run-openvswitch\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.873046    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5696ce-42fc-4253-b113-85471cef9d7f-var-lib-kubelet\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.874963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.873476    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef661a59-4e98-4d7b-8a57-02b0ec567c42-iptables-alerter-script\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.876540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.874852    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5696ce-42fc-4253-b113-85471cef9d7f-tmp\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.876540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.875215    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b19d6066-979b-4e85-9e24-6b20dc7ad427-agent-certs\") pod \"konnectivity-agent-8lfxt\" (UID: \"b19d6066-979b-4e85-9e24-6b20dc7ad427\") " pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:30.876540 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.875314    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5696ce-42fc-4253-b113-85471cef9d7f-etc-tuned\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.884231 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.884138    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6llk\" (UniqueName: \"kubernetes.io/projected/9cc40af6-5278-491d-83d0-7c61cc70943c-kube-api-access-p6llk\") pod \"multus-additional-cni-plugins-xnrtk\" (UID: \"9cc40af6-5278-491d-83d0-7c61cc70943c\") " pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:30.889375 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.889346    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrnv\" (UniqueName: \"kubernetes.io/projected/d1b5fdf1-1f55-47e4-a91a-909df54d83bb-kube-api-access-pfrnv\") pod \"ovnkube-node-6m2cv\" (UID: \"d1b5fdf1-1f55-47e4-a91a-909df54d83bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.889537 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.889518    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbqf\" (UniqueName: \"kubernetes.io/projected/d3c92389-4b78-4df8-88c7-be27ef0ce74e-kube-api-access-rwbqf\") pod \"aws-ebs-csi-driver-node-dtzs6\" (UID: \"d3c92389-4b78-4df8-88c7-be27ef0ce74e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:30.889910 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.889885    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9k5\" (UniqueName: \"kubernetes.io/projected/8d5696ce-42fc-4253-b113-85471cef9d7f-kube-api-access-pg9k5\") pod \"tuned-bgpfl\" (UID: \"8d5696ce-42fc-4253-b113-85471cef9d7f\") " pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:30.890568 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.890331    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvpb\" (UniqueName: \"kubernetes.io/projected/22afd025-a356-4e25-a21b-b1d8cd06dc7c-kube-api-access-jfvpb\") pod \"node-ca-j8zcx\" (UID: \"22afd025-a356-4e25-a21b-b1d8cd06dc7c\") " pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.890568 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.890375    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8qm\" (UniqueName: \"kubernetes.io/projected/075bcaa7-4039-43d5-8c2a-a654180769f9-kube-api-access-xk8qm\") pod \"multus-f6m68\" (UID: \"075bcaa7-4039-43d5-8c2a-a654180769f9\") " pod="openshift-multus/multus-f6m68"
Jul 08 16:33:30.890684 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.890622    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdhp\" (UniqueName: \"kubernetes.io/projected/ef661a59-4e98-4d7b-8a57-02b0ec567c42-kube-api-access-pxdhp\") pod \"iptables-alerter-lnd8n\" (UID: \"ef661a59-4e98-4d7b-8a57-02b0ec567c42\") " pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:30.891061 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.891040    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqv7w\" (UniqueName: \"kubernetes.io/projected/b11058bb-8eed-4973-b455-a320ededc11b-kube-api-access-fqv7w\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:30.960031 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.959744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dcpxh"
Jul 08 16:33:30.974114 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.973857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:30.977221 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:30.977185    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682cd642_9143_47ab_92ed_d33cfbd3eec9.slice/crio-c0eee4beb801c4665c6b29fc04452d49c90012e7ce6cefc4910853a7cc451a92 WatchSource:0}: Error finding container c0eee4beb801c4665c6b29fc04452d49c90012e7ce6cefc4910853a7cc451a92: Status 404 returned error can't find the container with id c0eee4beb801c4665c6b29fc04452d49c90012e7ce6cefc4910853a7cc451a92
Jul 08 16:33:30.985069 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.985045    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j8zcx"
Jul 08 16:33:30.991491 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:30.991311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bgpfl"
Jul 08 16:33:31.000278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.000247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f6m68"
Jul 08 16:33:31.002232 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.002197    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22afd025_a356_4e25_a21b_b1d8cd06dc7c.slice/crio-2997fde9570b185023aa20b1471f3809321a60099391c85c9b76fc682a2c4a00 WatchSource:0}: Error finding container 2997fde9570b185023aa20b1471f3809321a60099391c85c9b76fc682a2c4a00: Status 404 returned error can't find the container with id 2997fde9570b185023aa20b1471f3809321a60099391c85c9b76fc682a2c4a00
Jul 08 16:33:31.003428 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.003396    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5696ce_42fc_4253_b113_85471cef9d7f.slice/crio-26c6fa04ce4b1bad0ed3087b7e6660e5cb15ca2929d2c3b61542d9879d115356 WatchSource:0}: Error finding container 26c6fa04ce4b1bad0ed3087b7e6660e5cb15ca2929d2c3b61542d9879d115356: Status 404 returned error can't find the container with id 26c6fa04ce4b1bad0ed3087b7e6660e5cb15ca2929d2c3b61542d9879d115356
Jul 08 16:33:31.010045 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.010023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6"
Jul 08 16:33:31.012334 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.012304    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075bcaa7_4039_43d5_8c2a_a654180769f9.slice/crio-3ae226db7b2e1ea7e4ad6f70ab9db1ed9e4db6d4674bc2cbac589ac250182504 WatchSource:0}: Error finding container 3ae226db7b2e1ea7e4ad6f70ab9db1ed9e4db6d4674bc2cbac589ac250182504: Status 404 returned error can't find the container with id 3ae226db7b2e1ea7e4ad6f70ab9db1ed9e4db6d4674bc2cbac589ac250182504
Jul 08 16:33:31.019284 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.019239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnd8n"
Jul 08 16:33:31.029885 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.029848    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c92389_4b78_4df8_88c7_be27ef0ce74e.slice/crio-5b804b353c96094145f10f3d296c3842e761c31485efdd8e6853096477fb9a00 WatchSource:0}: Error finding container 5b804b353c96094145f10f3d296c3842e761c31485efdd8e6853096477fb9a00: Status 404 returned error can't find the container with id 5b804b353c96094145f10f3d296c3842e761c31485efdd8e6853096477fb9a00
Jul 08 16:33:31.030859 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.030767    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef661a59_4e98_4d7b_8a57_02b0ec567c42.slice/crio-e2ef283a941f9fe2cd0c7aec2a52159786e2426f06261c425cbc438adac2a9e0 WatchSource:0}: Error finding container e2ef283a941f9fe2cd0c7aec2a52159786e2426f06261c425cbc438adac2a9e0: Status 404 returned error can't find the container with id e2ef283a941f9fe2cd0c7aec2a52159786e2426f06261c425cbc438adac2a9e0
Jul 08 16:33:31.036197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.036173    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xnrtk"
Jul 08 16:33:31.043596 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.043572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:31.048829 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.048794    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc40af6_5278_491d_83d0_7c61cc70943c.slice/crio-96a5f551397573d30f914447d5d213ba9cd7cbf56c6ad114ee69438bed97248c WatchSource:0}: Error finding container 96a5f551397573d30f914447d5d213ba9cd7cbf56c6ad114ee69438bed97248c: Status 404 returned error can't find the container with id 96a5f551397573d30f914447d5d213ba9cd7cbf56c6ad114ee69438bed97248c
Jul 08 16:33:31.054992 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:33:31.054963    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19d6066_979b_4e85_9e24_6b20dc7ad427.slice/crio-cccc4a3b5aae44bed53edaef74031a29c51d49a01dbd6455190e7424c6a0b4c6 WatchSource:0}: Error finding container cccc4a3b5aae44bed53edaef74031a29c51d49a01dbd6455190e7424c6a0b4c6: Status 404 returned error can't find the container with id cccc4a3b5aae44bed53edaef74031a29c51d49a01dbd6455190e7424c6a0b4c6
Jul 08 16:33:31.276342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.276235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:31.276508 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.276432    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:31.276508 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.276464    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:31.276508 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.276479    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:31.276650 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.276555    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:32.276535466 +0000 UTC m=+4.215136718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:31.377271 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.377236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:31.377455 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.377300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:31.377455 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.377392    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:31.377455 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.377434    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:31.377622 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.377475    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:33:32.377452666 +0000 UTC m=+4.316053918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:31.377622 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:31.377500    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:32.37748909 +0000 UTC m=+4.316090345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:31.701192 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.701135    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:28:30 +0000 UTC" deadline="2028-03-09 02:53:27.085422415 +0000 UTC"
Jul 08 16:33:31.701192 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.701184    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14626h19m55.384241811s"
Jul 08 16:33:31.806593 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.806557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lfxt" event={"ID":"b19d6066-979b-4e85-9e24-6b20dc7ad427","Type":"ContainerStarted","Data":"cccc4a3b5aae44bed53edaef74031a29c51d49a01dbd6455190e7424c6a0b4c6"}
Jul 08 16:33:31.807778 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.807729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerStarted","Data":"96a5f551397573d30f914447d5d213ba9cd7cbf56c6ad114ee69438bed97248c"}
Jul 08 16:33:31.809127 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.809086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnd8n" event={"ID":"ef661a59-4e98-4d7b-8a57-02b0ec567c42","Type":"ContainerStarted","Data":"e2ef283a941f9fe2cd0c7aec2a52159786e2426f06261c425cbc438adac2a9e0"}
Jul 08 16:33:31.810381 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.810353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6m68" event={"ID":"075bcaa7-4039-43d5-8c2a-a654180769f9","Type":"ContainerStarted","Data":"3ae226db7b2e1ea7e4ad6f70ab9db1ed9e4db6d4674bc2cbac589ac250182504"}
Jul 08 16:33:31.811976 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.811948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bgpfl" event={"ID":"8d5696ce-42fc-4253-b113-85471cef9d7f","Type":"ContainerStarted","Data":"26c6fa04ce4b1bad0ed3087b7e6660e5cb15ca2929d2c3b61542d9879d115356"}
Jul 08 16:33:31.813276 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.813244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"177bd777249c9232a1d14d450fbe92ad4b894d68d37ce4ac953c6eb589c4f063"}
Jul 08 16:33:31.814461 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.814439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6" event={"ID":"d3c92389-4b78-4df8-88c7-be27ef0ce74e","Type":"ContainerStarted","Data":"5b804b353c96094145f10f3d296c3842e761c31485efdd8e6853096477fb9a00"}
Jul 08 16:33:31.815615 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.815585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j8zcx" event={"ID":"22afd025-a356-4e25-a21b-b1d8cd06dc7c","Type":"ContainerStarted","Data":"2997fde9570b185023aa20b1471f3809321a60099391c85c9b76fc682a2c4a00"}
Jul 08 16:33:31.817146 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:31.817090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dcpxh" event={"ID":"682cd642-9143-47ab-92ed-d33cfbd3eec9","Type":"ContainerStarted","Data":"c0eee4beb801c4665c6b29fc04452d49c90012e7ce6cefc4910853a7cc451a92"}
Jul 08 16:33:32.285128 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.285088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:32.285334 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.285306    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:32.285334 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.285328    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:32.285461 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.285342    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:32.285461 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.285427    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:34.285407154 +0000 UTC m=+6.224008425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:32.386457 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.386408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:32.386662 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.386513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:32.386662 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.386631    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:32.386662 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.386650    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:32.386854 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.386732    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:34.386691087 +0000 UTC m=+6.325292359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:32.386854 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.386756    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:33:34.386744945 +0000 UTC m=+6.325346201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:32.702203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.701950    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:28:30 +0000 UTC" deadline="2028-02-22 22:00:39.828447551 +0000 UTC"
Jul 08 16:33:32.702203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.702204    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14261h27m7.126250356s"
Jul 08 16:33:32.797559 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.796739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:32.797559 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.796909    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:32.797559 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.797404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:32.797559 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.797530    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:32.797914 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.797607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:32.797914 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:32.797684    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:32.834568 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.834406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal" event={"ID":"938243ce709c303fec6b51c38490eae8","Type":"ContainerStarted","Data":"72eaa60898571538dc6017a05b2b58ff35011743608ab381d5841856311bb4bb"}
Jul 08 16:33:32.863350 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:32.863276    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-233.ec2.internal" podStartSLOduration=2.863253384 podStartE2EDuration="2.863253384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:32.862851025 +0000 UTC m=+4.801452291" watchObservedRunningTime="2026-07-08 16:33:32.863253384 +0000 UTC m=+4.801854658"
Jul 08 16:33:33.853774 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:33.852400    2561 generic.go:356] "Generic (PLEG): container finished" podID="9a9307618ad481c92addb0219af76bb7" containerID="3c1588b3ee149e3dd75abf237fc2c0317f6bd1059390cf78025042f186623c68" exitCode=0
Jul 08 16:33:33.853774 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:33.853472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal" event={"ID":"9a9307618ad481c92addb0219af76bb7","Type":"ContainerDied","Data":"3c1588b3ee149e3dd75abf237fc2c0317f6bd1059390cf78025042f186623c68"}
Jul 08 16:33:34.305623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:34.305534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:34.305888 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.305770    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:34.305888 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.305790    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:34.305888 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.305803    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:34.305888 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.305867    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:38.305848289 +0000 UTC m=+10.244449541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:34.406859 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:34.406820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:34.407035 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:34.406910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:34.407100 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.407036    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:34.407100 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.407097    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:33:38.407078681 +0000 UTC m=+10.345679945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:34.407504 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.407482    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:34.407609 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.407539    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:38.407524466 +0000 UTC m=+10.346125728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:34.798355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:34.797803    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:34.798355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:34.797817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:34.798355 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.797947    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:34.798355 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.798034    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:34.798355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:34.798125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:34.798355 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:34.798232    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:36.796590 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:36.796557    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:36.797029 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:36.796702    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:36.797029 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:36.796964    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:36.797155 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:36.797037    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:36.797155 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:36.797144    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:36.797252 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:36.797220    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:38.340800 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:38.340737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:38.341199 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.341129    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:38.341199 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.341160    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:38.341199 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.341173    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:38.341397 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.341336    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:46.341263381 +0000 UTC m=+18.279864629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:38.442336 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:38.442272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:38.442524 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:38.442382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:38.442524 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.442417    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:38.442524 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.442487    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:33:46.442471497 +0000 UTC m=+18.381072749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:38.442524 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.442485    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:38.442524 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.442525    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:33:46.442514706 +0000 UTC m=+18.381115957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:38.797803 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:38.797765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:38.797986 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.797900    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:38.798366 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:38.798346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:38.798603 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.798464    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:38.798751 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:38.798629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:38.798828 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:38.798745    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:40.796776 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:40.796747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:40.797229 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:40.796862    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:40.797229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:40.796994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:40.797229 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:40.797109    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:40.797229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:40.797141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:40.797229 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:40.797194    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:42.796549 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:42.796514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:42.796549 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:42.796539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:42.797062 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:42.796514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:42.797062 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:42.796653    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:42.797062 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:42.796789    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:42.797062 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:42.796874    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:44.796324 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:44.796282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:44.796823 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:44.796281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:44.796823 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:44.796422    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:44.796823 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:44.796281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:44.796823 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:44.796496    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:44.796823 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:44.796529    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:46.409924 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:46.409890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:46.410365 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.410074    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:33:46.410365 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.410108    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:33:46.410365 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.410120    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:46.410365 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.410180    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:02.410165898 +0000 UTC m=+34.348767151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:33:46.511109 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:46.511069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:46.511290 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:46.511129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:46.511290 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.511282    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:46.511395 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.511326    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:46.511395 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.511354    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:02.51133436 +0000 UTC m=+34.449935627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:33:46.511395 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.511394    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:02.511374997 +0000 UTC m=+34.449976287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:33:46.797023 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:46.796988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:46.797207 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:46.796989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:46.797207 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.797154    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:46.797207 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:46.796989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:46.797323 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.797198    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:46.797323 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:46.797287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:48.796850 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:48.796809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:48.797301 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:48.796892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:48.797301 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:48.796905    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:48.797301 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:48.797003    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:48.797301 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:48.797018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:48.797301 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:48.797120    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:50.797188 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.796877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:50.798169 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.796877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:50.798169 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:50.797296    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:50.798169 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.796874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:50.798169 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:50.797390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:50.798169 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:50.797436    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:50.886534 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.886503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal" event={"ID":"9a9307618ad481c92addb0219af76bb7","Type":"ContainerStarted","Data":"f6fae4bba192b6ffefdcfb0a61951949eba003b850baab9149b0b72ea65efbcd"}
Jul 08 16:33:50.888055 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.888031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lfxt" event={"ID":"b19d6066-979b-4e85-9e24-6b20dc7ad427","Type":"ContainerStarted","Data":"f8a5a9bfd4ff9ab6a4eb33094f643bf231a0bb4396a56a42da909244926bfd61"}
Jul 08 16:33:50.889438 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.889413    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cc40af6-5278-491d-83d0-7c61cc70943c" containerID="2b3f969fe95fcebd283e20cb054fa7cc17e20c168c64049177d2642c4c993c48" exitCode=0
Jul 08 16:33:50.889555 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.889496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerDied","Data":"2b3f969fe95fcebd283e20cb054fa7cc17e20c168c64049177d2642c4c993c48"}
Jul 08 16:33:50.890965 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.890944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f6m68" event={"ID":"075bcaa7-4039-43d5-8c2a-a654180769f9","Type":"ContainerStarted","Data":"42a108384918808d3fac3567d5fdd02cff938e8837e33d6a893222d3c882c03f"}
Jul 08 16:33:50.892336 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.892309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bgpfl" event={"ID":"8d5696ce-42fc-4253-b113-85471cef9d7f","Type":"ContainerStarted","Data":"5698e417828bf8e753843f08e54ba565320a7c5aec82fbaeed6514d8ce742aad"}
Jul 08 16:33:50.894720 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.894703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"d9daf2eb50495f797cba800ea43027114425c850cbc6951b73dfc90a2463ee0e"}
Jul 08 16:33:50.894804 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.894752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"beaa57863e840f2c3f24362dd71b130cae5334f41c6f73cb47ebd98c0c3b7a72"}
Jul 08 16:33:50.894848 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.894820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"96f72eed2f18310f2b9b1eeb41402e1c1ec952626f5591073ed8402e0027b820"}
Jul 08 16:33:50.894885 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.894848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"ef825cf4e86e070ed4af649a1aef574d0dfb701c31e4cf0e1f96aeeabfc3a25a"}
Jul 08 16:33:50.896053 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.896035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6" event={"ID":"d3c92389-4b78-4df8-88c7-be27ef0ce74e","Type":"ContainerStarted","Data":"39facbd3b3cef873ef5825fbb8719c98b5b52d5745d2968a6bbbce164003cc88"}
Jul 08 16:33:50.897286 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.897264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j8zcx" event={"ID":"22afd025-a356-4e25-a21b-b1d8cd06dc7c","Type":"ContainerStarted","Data":"cce563c561511b6ffbe2a14a83ad972b8e167257022c71a90e8ef849e6a2257d"}
Jul 08 16:33:50.898675 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.898654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dcpxh" event={"ID":"682cd642-9143-47ab-92ed-d33cfbd3eec9","Type":"ContainerStarted","Data":"96260daa51d53b1edc6197901ed7f36aa2bd86e0b841f7f7956a53c03d2b6fc1"}
Jul 08 16:33:50.917512 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.917416    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-233.ec2.internal" podStartSLOduration=20.91740092 podStartE2EDuration="20.91740092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:50.910827602 +0000 UTC m=+22.849428873" watchObservedRunningTime="2026-07-08 16:33:50.91740092 +0000 UTC m=+22.856002189"
Jul 08 16:33:50.977802 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:50.977754    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f6m68" podStartSLOduration=2.580965589 podStartE2EDuration="21.977737495s" totalImagesPullingTime="19.396771906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:50.950271382 +0000 UTC m=+22.888872651" watchObservedRunningTime="2026-07-08 16:33:50.977737495 +0000 UTC m=+22.916338764"
Jul 08 16:33:51.007378 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.007332    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dcpxh" podStartSLOduration=2.58161548 podStartE2EDuration="22.007315304s" totalImagesPullingTime="19.425699824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:50.97766811 +0000 UTC m=+22.916269391" watchObservedRunningTime="2026-07-08 16:33:51.007315304 +0000 UTC m=+22.945916573"
Jul 08 16:33:51.007753 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.007722    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j8zcx" podStartSLOduration=11.138317532 podStartE2EDuration="22.007713209s" totalImagesPullingTime="10.869395677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:51.007289451 +0000 UTC m=+22.945890721" watchObservedRunningTime="2026-07-08 16:33:51.007713209 +0000 UTC m=+22.946314478"
Jul 08 16:33:51.057675 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.057628    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bgpfl" podStartSLOduration=2.65509636 podStartE2EDuration="22.057612443s" totalImagesPullingTime="19.402516083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:51.055990981 +0000 UTC m=+22.994592252" watchObservedRunningTime="2026-07-08 16:33:51.057612443 +0000 UTC m=+22.996213712"
Jul 08 16:33:51.902326 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.902290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnd8n" event={"ID":"ef661a59-4e98-4d7b-8a57-02b0ec567c42","Type":"ContainerStarted","Data":"ae4ee026b9e4e370b3260dea0e4ddce7f4607bd64298ab0df8a50ce6550e897a"}
Jul 08 16:33:51.905864 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.905833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"22763c5f368617e481fc9368c2684a4acbfa3e270f3110b7d79f396b1a4369e0"}
Jul 08 16:33:51.906012 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.905913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"30a3ed8ef4e69818aa46185f236dadd2994765972b63b2628d5c287b81252ede"}
Jul 08 16:33:51.920356 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:51.920299    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8lfxt" podStartSLOduration=3.56911957 podStartE2EDuration="22.920278849s" totalImagesPullingTime="19.351159279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:51.120849331 +0000 UTC m=+23.059450600" watchObservedRunningTime="2026-07-08 16:33:51.920278849 +0000 UTC m=+23.858880120"
Jul 08 16:33:52.503415 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.503385    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:33:52.724287 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.724179    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:33:52.503406111Z","UUID":"b1731942-2eaf-462b-b55a-01326ae40ed1","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:33:52.730365 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.726846    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:33:52.730365 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.726902    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:33:52.797122 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.797083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:52.797296 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.797127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:52.797296 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.797086    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:52.797296 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:52.797238    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:52.797456 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:52.797351    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:52.797456 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:52.797431    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:52.893305 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.893265    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:52.893900 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.893882    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:52.909889 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.909858    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6" event={"ID":"d3c92389-4b78-4df8-88c7-be27ef0ce74e","Type":"ContainerStarted","Data":"f7fb9db7ea293a93a84f9517a829a574b7d07f9116e0b188699cd160e0a42a11"}
Jul 08 16:33:52.950984 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:52.950933    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lnd8n" podStartSLOduration=4.5766894 podStartE2EDuration="23.950914345s" totalImagesPullingTime="19.374224945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:51.920176909 +0000 UTC m=+23.858778181" watchObservedRunningTime="2026-07-08 16:33:52.950914345 +0000 UTC m=+24.889515615"
Jul 08 16:33:53.916299 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:53.916274    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:33:53.916804 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:53.916306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"297e6d8db83495d1854a617170c229c3a8726d17edcd8d52b7fcf46f868642a3"}
Jul 08 16:33:54.797186 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:54.797013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:54.797352 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:54.797019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:54.797352 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:54.797266    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:54.797352 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:54.797022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:54.797352 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:54.797341    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:54.797551 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:54.797395    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:54.921009 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:54.920978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6" event={"ID":"d3c92389-4b78-4df8-88c7-be27ef0ce74e","Type":"ContainerStarted","Data":"c93c167a300d938a3634b1397de94274f8e6b0a49e4376a066733975561577c3"}
Jul 08 16:33:54.941046 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:54.941004    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dtzs6" podStartSLOduration=2.705764576 podStartE2EDuration="25.940987522s" totalImagesPullingTime="23.235222946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:54.940477983 +0000 UTC m=+26.879079254" watchObservedRunningTime="2026-07-08 16:33:54.940987522 +0000 UTC m=+26.879588791"
Jul 08 16:33:55.926882 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:55.926843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" event={"ID":"d1b5fdf1-1f55-47e4-a91a-909df54d83bb","Type":"ContainerStarted","Data":"6722a2aa38b0e50a027be67658adccdff1fbd973f920ef14ce4db0081bfcde94"}
Jul 08 16:33:55.957526 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:55.957467    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv" podStartSLOduration=7.432781362 podStartE2EDuration="26.957451495s" totalImagesPullingTime="19.524670133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:33:55.955994313 +0000 UTC m=+27.894595584" watchObservedRunningTime="2026-07-08 16:33:55.957451495 +0000 UTC m=+27.896052764"
Jul 08 16:33:55.999350 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:55.999310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:55.999513 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:55.999454    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:33:56.000075 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.000042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8lfxt"
Jul 08 16:33:56.797041 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.796814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:56.797233 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.796816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:56.797233 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:56.797137    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:56.797345 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:56.797235    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:56.797345 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.796822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:56.797345 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:56.797333    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:56.930205 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.930173    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:56.930658 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.930264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:56.930658 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.930314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:56.946776 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.946751    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:56.947037 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:56.946979    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:33:58.797865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:58.797841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:33:58.798213 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:58.797915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:33:58.798213 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:58.797932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:33:58.798213 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:58.798019    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:33:58.798213 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:58.798053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:33:58.798213 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:33:58.798133    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:33:58.935062 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:58.935035    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cc40af6-5278-491d-83d0-7c61cc70943c" containerID="3a43dc9eefee497c9e3f00b73d4b51c14266e7e0431a3663cc0be494b56659c1" exitCode=0
Jul 08 16:33:58.935233 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:58.935112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerDied","Data":"3a43dc9eefee497c9e3f00b73d4b51c14266e7e0431a3663cc0be494b56659c1"}
Jul 08 16:33:59.938913 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:59.938878    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cc40af6-5278-491d-83d0-7c61cc70943c" containerID="506b4ae6bd2274348cb02a661ee1afaa049007a5aa5aeaa617dc512380afaa67" exitCode=0
Jul 08 16:33:59.939253 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:33:59.938936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerDied","Data":"506b4ae6bd2274348cb02a661ee1afaa049007a5aa5aeaa617dc512380afaa67"}
Jul 08 16:34:00.796297 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:00.796267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:00.796450 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:00.796267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:00.796450 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:00.796364    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:00.796450 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:00.796378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:00.796554 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:00.796451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:00.796554 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:00.796524    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:01.945601 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:01.945570    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cc40af6-5278-491d-83d0-7c61cc70943c" containerID="d4674389a438e3c32838f633a53bca9b766e998ad5996376f038499e1639b33d" exitCode=0
Jul 08 16:34:01.945978 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:01.945622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerDied","Data":"d4674389a438e3c32838f633a53bca9b766e998ad5996376f038499e1639b33d"}
Jul 08 16:34:02.429997 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:02.429955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:02.430313 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.430119    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:02.430313 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.430145    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:02.430313 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.430156    2561 projected.go:196] Error preparing data for projected volume kube-api-access-vts45 for pod openshift-network-diagnostics/network-check-target-ddnsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:02.430313 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.430205    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45 podName:9a6f26ab-7d25-4a7c-b527-d7a717586e35 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:34.43019064 +0000 UTC m=+66.368791893 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vts45" (UniqueName: "kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45") pod "network-check-target-ddnsp" (UID: "9a6f26ab-7d25-4a7c-b527-d7a717586e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:02.530506 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:02.530464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:02.530667 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:02.530543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:02.530667 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.530619    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:02.530667 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.530657    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:02.530866 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.530710    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs podName:b11058bb-8eed-4973-b455-a320ededc11b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:34.530675361 +0000 UTC m=+66.469276614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs") pod "network-metrics-daemon-pdp4m" (UID: "b11058bb-8eed-4973-b455-a320ededc11b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:02.530866 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.530727    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret podName:03e71b89-45d3-415c-844d-c81aa15d87f7 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:34.530719528 +0000 UTC m=+66.469320775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret") pod "global-pull-secret-syncer-dlb5z" (UID: "03e71b89-45d3-415c-844d-c81aa15d87f7") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:02.796350 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:02.796270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:02.796496 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:02.796282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:02.796496 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.796413    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:02.796496 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:02.796282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:02.796496 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.796478    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:02.796633 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:02.796557    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:04.797059 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:04.797019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:04.797531 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:04.797019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:04.797531 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:04.797170    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:04.797531 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:04.797019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:04.797531 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:04.797208    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:04.797531 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:04.797287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:06.796213 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:06.796184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:06.796706 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:06.796187    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:06.796706 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:06.796184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:06.796706 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:06.796309    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:06.796706 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:06.796388    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:06.796706 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:06.796490    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:07.962585 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:07.962549    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cc40af6-5278-491d-83d0-7c61cc70943c" containerID="4348373f3ed8688a04d7624ca1acce3a015255c6bc66741ae6e6716623ec8999" exitCode=0
Jul 08 16:34:07.962995 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:07.962618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerDied","Data":"4348373f3ed8688a04d7624ca1acce3a015255c6bc66741ae6e6716623ec8999"}
Jul 08 16:34:08.797189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:08.797152    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:08.797388 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:08.797243    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:08.797388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:08.797258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:08.797388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:08.797285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:08.797388 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:08.797347    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:08.797601 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:08.797404    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:08.967638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:08.967602    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cc40af6-5278-491d-83d0-7c61cc70943c" containerID="e731c4c0033192366a20ab0bc2402d245d7d1fc7e203b6d66b489524bb00b517" exitCode=0
Jul 08 16:34:08.968011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:08.967667    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerDied","Data":"e731c4c0033192366a20ab0bc2402d245d7d1fc7e203b6d66b489524bb00b517"}
Jul 08 16:34:09.973724 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:09.973678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" event={"ID":"9cc40af6-5278-491d-83d0-7c61cc70943c","Type":"ContainerStarted","Data":"90c7108e55cfeece4e7732655916bea32afbeb2b1e2a60d9d9ab8cb37d99193c"}
Jul 08 16:34:10.000300 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:10.000252    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xnrtk" podStartSLOduration=6.436582062 podStartE2EDuration="41.000234536s" totalImagesPullingTime="34.563652474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:10.000106615 +0000 UTC m=+41.938707906" watchObservedRunningTime="2026-07-08 16:34:10.000234536 +0000 UTC m=+41.938835805"
Jul 08 16:34:10.797019 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:10.796986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:10.797191 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:10.796987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:10.797191 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:10.797090    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:10.797191 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:10.796996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:10.797288 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:10.797175    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:10.797288 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:10.797265    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:12.796208 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:12.796173    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:12.796561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:12.796173    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:12.796561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:12.796302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:12.796561 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:12.796299    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:12.796561 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:12.796385    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:12.796561 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:12.796490    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:13.342938 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:13.342660    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdp4m"]
Jul 08 16:34:13.343091 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:13.343052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:13.343199 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:13.343175    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:13.343607 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:13.343587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddnsp"]
Jul 08 16:34:13.343696 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:13.343668    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:13.343782 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:13.343764    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:13.351822 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:13.351795    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dlb5z"]
Jul 08 16:34:13.352034 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:13.352019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:13.352136 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:13.352115    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:14.796803 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:14.796761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:14.796803 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:14.796788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:14.797440 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:14.796761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:14.797440 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:14.796883    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:14.797440 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:14.796977    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:14.797440 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:14.797058    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:16.796431 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:16.796402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:16.796431 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:16.796438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:16.796991 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:16.796404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:16.796991 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:16.796531    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ddnsp" podUID="9a6f26ab-7d25-4a7c-b527-d7a717586e35"
Jul 08 16:34:16.796991 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:16.796626    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dlb5z" podUID="03e71b89-45d3-415c-844d-c81aa15d87f7"
Jul 08 16:34:16.796991 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:16.796747    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdp4m" podUID="b11058bb-8eed-4973-b455-a320ededc11b"
Jul 08 16:34:17.442377 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.442345    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-233.ec2.internal" event="NodeReady"
Jul 08 16:34:17.442544 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.442457    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:34:17.517052 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.517014    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-czxrs"]
Jul 08 16:34:17.528374 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.528343    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2f8x4"]
Jul 08 16:34:17.528543 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.528466    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.532139 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.532099    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s4gk8\""
Jul 08 16:34:17.533224 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.532861    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:34:17.533224 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.533085    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:34:17.537098 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.537069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2f8x4"]
Jul 08 16:34:17.537252 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.537235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-czxrs"]
Jul 08 16:34:17.537252 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.537265    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j58g5"]
Jul 08 16:34:17.537437 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.537187    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.540230 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.540210    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:34:17.540308 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.540258    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:34:17.540356 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.540260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twqsl\""
Jul 08 16:34:17.540356 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.540337    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:34:17.549451 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.549435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.552313 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.552293    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:34:17.552415 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.552338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:34:17.552415 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.552368    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mg5k6\""
Jul 08 16:34:17.552633 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.552610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:34:17.552870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.552853    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:34:17.553210 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.553195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j58g5"]
Jul 08 16:34:17.635699 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.635641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33f300c-dab4-44c7-9341-cea20d6fbd2c-metrics-tls\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.635894 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.635714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d33f300c-dab4-44c7-9341-cea20d6fbd2c-config-volume\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.635894 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.635750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33f300c-dab4-44c7-9341-cea20d6fbd2c-tmp-dir\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.635894 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.635767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhkd\" (UniqueName: \"kubernetes.io/projected/d33f300c-dab4-44c7-9341-cea20d6fbd2c-kube-api-access-swhkd\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.635894 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.635791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxnc\" (UniqueName: \"kubernetes.io/projected/0207049b-d24a-4e85-a473-97a5b8587bdf-kube-api-access-mgxnc\") pod \"ingress-canary-2f8x4\" (UID: \"0207049b-d24a-4e85-a473-97a5b8587bdf\") " pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.635894 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.635876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0207049b-d24a-4e85-a473-97a5b8587bdf-cert\") pod \"ingress-canary-2f8x4\" (UID: \"0207049b-d24a-4e85-a473-97a5b8587bdf\") " pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.736705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d33f300c-dab4-44c7-9341-cea20d6fbd2c-config-volume\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.736705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6p5f\" (UniqueName: \"kubernetes.io/projected/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-kube-api-access-k6p5f\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.736705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33f300c-dab4-44c7-9341-cea20d6fbd2c-tmp-dir\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.736705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhkd\" (UniqueName: \"kubernetes.io/projected/d33f300c-dab4-44c7-9341-cea20d6fbd2c-kube-api-access-swhkd\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.737006 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxnc\" (UniqueName: \"kubernetes.io/projected/0207049b-d24a-4e85-a473-97a5b8587bdf-kube-api-access-mgxnc\") pod \"ingress-canary-2f8x4\" (UID: \"0207049b-d24a-4e85-a473-97a5b8587bdf\") " pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.737006 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-data-volume\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.737006 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.737006 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0207049b-d24a-4e85-a473-97a5b8587bdf-cert\") pod \"ingress-canary-2f8x4\" (UID: \"0207049b-d24a-4e85-a473-97a5b8587bdf\") " pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.737006 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.736932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-crio-socket\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.737231 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.737019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.737231 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.737065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33f300c-dab4-44c7-9341-cea20d6fbd2c-metrics-tls\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.737231 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.737159    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33f300c-dab4-44c7-9341-cea20d6fbd2c-tmp-dir\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.737363 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.737329    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d33f300c-dab4-44c7-9341-cea20d6fbd2c-config-volume\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.740861 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.740835    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33f300c-dab4-44c7-9341-cea20d6fbd2c-metrics-tls\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.740989 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.740875    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0207049b-d24a-4e85-a473-97a5b8587bdf-cert\") pod \"ingress-canary-2f8x4\" (UID: \"0207049b-d24a-4e85-a473-97a5b8587bdf\") " pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.745149 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.745127    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhkd\" (UniqueName: \"kubernetes.io/projected/d33f300c-dab4-44c7-9341-cea20d6fbd2c-kube-api-access-swhkd\") pod \"dns-default-czxrs\" (UID: \"d33f300c-dab4-44c7-9341-cea20d6fbd2c\") " pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.745319 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.745303    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxnc\" (UniqueName: \"kubernetes.io/projected/0207049b-d24a-4e85-a473-97a5b8587bdf-kube-api-access-mgxnc\") pod \"ingress-canary-2f8x4\" (UID: \"0207049b-d24a-4e85-a473-97a5b8587bdf\") " pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.838309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-data-volume\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-crio-socket\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6p5f\" (UniqueName: \"kubernetes.io/projected/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-kube-api-access-k6p5f\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838714    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-data-volume\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838730    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-crio-socket\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.838930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.838919    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.839988 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.839974    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:17.840758 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.840738    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.846472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.846439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2f8x4"
Jul 08 16:34:17.847226 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.847207    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6p5f\" (UniqueName: \"kubernetes.io/projected/fcc2ec40-2691-48c8-88df-a6f7e67ee07c-kube-api-access-k6p5f\") pod \"insights-runtime-extractor-j58g5\" (UID: \"fcc2ec40-2691-48c8-88df-a6f7e67ee07c\") " pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:17.859358 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:17.859270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j58g5"
Jul 08 16:34:18.044229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.044050    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2f8x4"]
Jul 08 16:34:18.047185 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:18.047160    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0207049b_d24a_4e85_a473_97a5b8587bdf.slice/crio-74c4b6e243321a6320ee7039b9c9e0517fa6d635a6685966f5159be3206f114c WatchSource:0}: Error finding container 74c4b6e243321a6320ee7039b9c9e0517fa6d635a6685966f5159be3206f114c: Status 404 returned error can't find the container with id 74c4b6e243321a6320ee7039b9c9e0517fa6d635a6685966f5159be3206f114c
Jul 08 16:34:18.052789 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.052763    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-czxrs"]
Jul 08 16:34:18.053506 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.053432    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j58g5"]
Jul 08 16:34:18.056210 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:18.056188    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc2ec40_2691_48c8_88df_a6f7e67ee07c.slice/crio-67c7f02e3aafb58651f988645d6f47f0b5f633236b07eeacd04d51d964cdf24f WatchSource:0}: Error finding container 67c7f02e3aafb58651f988645d6f47f0b5f633236b07eeacd04d51d964cdf24f: Status 404 returned error can't find the container with id 67c7f02e3aafb58651f988645d6f47f0b5f633236b07eeacd04d51d964cdf24f
Jul 08 16:34:18.056862 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:18.056844    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33f300c_dab4_44c7_9341_cea20d6fbd2c.slice/crio-ab11716b88e289822c2de21a1be8c23b136926304932db540e217fa12acb7cec WatchSource:0}: Error finding container ab11716b88e289822c2de21a1be8c23b136926304932db540e217fa12acb7cec: Status 404 returned error can't find the container with id ab11716b88e289822c2de21a1be8c23b136926304932db540e217fa12acb7cec
Jul 08 16:34:18.058676 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.058650    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"]
Jul 08 16:34:18.076840 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.076813    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"]
Jul 08 16:34:18.076939 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.076880    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.079705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.079673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9l4nm\""
Jul 08 16:34:18.079807 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.079673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 08 16:34:18.079870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.079816    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:34:18.080132 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.080115    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 08 16:34:18.080288 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.080273    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:34:18.083903 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.080965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:34:18.241369 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.241327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25qd\" (UniqueName: \"kubernetes.io/projected/dc9e4e10-677d-4ad7-a326-ede65aa451b8-kube-api-access-b25qd\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.241369 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.241372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9e4e10-677d-4ad7-a326-ede65aa451b8-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.241622 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.241394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc9e4e10-677d-4ad7-a326-ede65aa451b8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.241622 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.241522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc9e4e10-677d-4ad7-a326-ede65aa451b8-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.342215 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.342122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25qd\" (UniqueName: \"kubernetes.io/projected/dc9e4e10-677d-4ad7-a326-ede65aa451b8-kube-api-access-b25qd\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.342215 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.342163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9e4e10-677d-4ad7-a326-ede65aa451b8-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.342215 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.342184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc9e4e10-677d-4ad7-a326-ede65aa451b8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.342215 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.342217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc9e4e10-677d-4ad7-a326-ede65aa451b8-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.342935 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.342914    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc9e4e10-677d-4ad7-a326-ede65aa451b8-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.345323 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.345298    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc9e4e10-677d-4ad7-a326-ede65aa451b8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.345418 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.345368    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc9e4e10-677d-4ad7-a326-ede65aa451b8-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.350442 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.350424    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25qd\" (UniqueName: \"kubernetes.io/projected/dc9e4e10-677d-4ad7-a326-ede65aa451b8-kube-api-access-b25qd\") pod \"prometheus-operator-567f7b8b7c-mzbsx\" (UID: \"dc9e4e10-677d-4ad7-a326-ede65aa451b8\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.389332 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.389300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"
Jul 08 16:34:18.529760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.529668    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx"]
Jul 08 16:34:18.532463 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:18.532434    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9e4e10_677d_4ad7_a326_ede65aa451b8.slice/crio-754824ca37f641007cb7642436a684ec4a7f39bd07f8829002549aa01cd507d5 WatchSource:0}: Error finding container 754824ca37f641007cb7642436a684ec4a7f39bd07f8829002549aa01cd507d5: Status 404 returned error can't find the container with id 754824ca37f641007cb7642436a684ec4a7f39bd07f8829002549aa01cd507d5
Jul 08 16:34:18.797808 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.797776    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:18.798027 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.797866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:18.798027 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.797913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:18.801328 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.801199    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:34:18.802659 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.802636    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s7kjq\""
Jul 08 16:34:18.802932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.802885    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:34:18.803095 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.803078    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:34:18.803297 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.803278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4cx7\""
Jul 08 16:34:18.803518 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:18.803467    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:34:19.001844 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:19.001794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j58g5" event={"ID":"fcc2ec40-2691-48c8-88df-a6f7e67ee07c","Type":"ContainerStarted","Data":"67c7f02e3aafb58651f988645d6f47f0b5f633236b07eeacd04d51d964cdf24f"}
Jul 08 16:34:19.003710 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:19.003616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2f8x4" event={"ID":"0207049b-d24a-4e85-a473-97a5b8587bdf","Type":"ContainerStarted","Data":"74c4b6e243321a6320ee7039b9c9e0517fa6d635a6685966f5159be3206f114c"}
Jul 08 16:34:19.005372 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:19.005303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czxrs" event={"ID":"d33f300c-dab4-44c7-9341-cea20d6fbd2c","Type":"ContainerStarted","Data":"ab11716b88e289822c2de21a1be8c23b136926304932db540e217fa12acb7cec"}
Jul 08 16:34:19.006797 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:19.006773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx" event={"ID":"dc9e4e10-677d-4ad7-a326-ede65aa451b8","Type":"ContainerStarted","Data":"754824ca37f641007cb7642436a684ec4a7f39bd07f8829002549aa01cd507d5"}
Jul 08 16:34:25.028054 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.028021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j58g5" event={"ID":"fcc2ec40-2691-48c8-88df-a6f7e67ee07c","Type":"ContainerStarted","Data":"6c5c9f157567c3d1d3794d08ef0bd6645408055141749f78c244fda7e481bd34"}
Jul 08 16:34:25.028517 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.028077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j58g5" event={"ID":"fcc2ec40-2691-48c8-88df-a6f7e67ee07c","Type":"ContainerStarted","Data":"9d718babfa174ace2da597ea7d3788b8418ef4e3e01fc497012de26521864ba6"}
Jul 08 16:34:25.029532 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.029509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2f8x4" event={"ID":"0207049b-d24a-4e85-a473-97a5b8587bdf","Type":"ContainerStarted","Data":"8a3fc32c943136cbd247d3a72ddee8f4a68e4ecb74519859ef12a6c12afbd16d"}
Jul 08 16:34:25.031351 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.031329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czxrs" event={"ID":"d33f300c-dab4-44c7-9341-cea20d6fbd2c","Type":"ContainerStarted","Data":"b9e2725d72816e4d6b2ecf0006aaece6748907fe3569ee6372fb4999cec054c4"}
Jul 08 16:34:25.031447 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.031368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czxrs" event={"ID":"d33f300c-dab4-44c7-9341-cea20d6fbd2c","Type":"ContainerStarted","Data":"ee448c6a6b4f38641a3c03432524b7fe92e7fc73d936d1d876a4579617545c95"}
Jul 08 16:34:25.031447 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.031391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:25.033033 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.033011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx" event={"ID":"dc9e4e10-677d-4ad7-a326-ede65aa451b8","Type":"ContainerStarted","Data":"101a2c3c9948fd08ed4a167e97f418942dd94db94b4ab3af9f34eba9c64eb98c"}
Jul 08 16:34:25.033120 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.033057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx" event={"ID":"dc9e4e10-677d-4ad7-a326-ede65aa451b8","Type":"ContainerStarted","Data":"874170901fe53e1084b21d657ebe07edddec581f9cd9929b2322813d4b4d017e"}
Jul 08 16:34:25.046893 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.046852    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2f8x4" podStartSLOduration=2.066733375 podStartE2EDuration="8.04684052s" totalImagesPullingTime="5.980107145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:25.046106015 +0000 UTC m=+56.984707287" watchObservedRunningTime="2026-07-08 16:34:25.04684052 +0000 UTC m=+56.985441789"
Jul 08 16:34:25.064851 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.064773    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-czxrs" podStartSLOduration=2.095792933 podStartE2EDuration="8.064760111s" totalImagesPullingTime="5.968967178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:25.064047036 +0000 UTC m=+57.002648306" watchObservedRunningTime="2026-07-08 16:34:25.064760111 +0000 UTC m=+57.003361380"
Jul 08 16:34:25.081781 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:25.081735    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-mzbsx" podStartSLOduration=1.5868350759999998 podStartE2EDuration="7.081721632s" totalImagesPullingTime="5.494886556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:25.081120082 +0000 UTC m=+57.019721349" watchObservedRunningTime="2026-07-08 16:34:25.081721632 +0000 UTC m=+57.020322879"
Jul 08 16:34:27.404984 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.404961    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"]
Jul 08 16:34:27.408013 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.407993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.410893 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.410872    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 08 16:34:27.411007 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.410915    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 08 16:34:27.411204 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.411189    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vx7cq\""
Jul 08 16:34:27.416806 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.416787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad20774-3ad0-4942-a467-66e8da1044fe-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.416902 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.416811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lw2\" (UniqueName: \"kubernetes.io/projected/6ad20774-3ad0-4942-a467-66e8da1044fe-kube-api-access-p5lw2\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.416902 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.416851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.417009 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.416976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.420399 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.420379    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"]
Jul 08 16:34:27.460745 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.460713    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ctcsm"]
Jul 08 16:34:27.464122 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.464107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.467261 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.467241    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:34:27.469335 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.469317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:34:27.469533 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.469519    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-klr5s\""
Jul 08 16:34:27.469603 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.469545    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:34:27.517889 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.517855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25608786-14b6-429f-912c-1ca46206e171-node-exporter-accelerators-collector-config\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.517889 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.517888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25608786-14b6-429f-912c-1ca46206e171-metrics-client-ca\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518094 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.517911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad20774-3ad0-4942-a467-66e8da1044fe-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.518094 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.517928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lw2\" (UniqueName: \"kubernetes.io/projected/6ad20774-3ad0-4942-a467-66e8da1044fe-kube-api-access-p5lw2\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.518094 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.517995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-root\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25608786-14b6-429f-912c-1ca46206e171-node-exporter-textfile\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25608786-14b6-429f-912c-1ca46206e171-node-exporter-tls\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.518309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25608786-14b6-429f-912c-1ca46206e171-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-sys\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.518446 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvxw\" (UniqueName: \"kubernetes.io/projected/25608786-14b6-429f-912c-1ca46206e171-kube-api-access-jnvxw\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518446 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-node-exporter-wtmp\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.518446 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:27.518375    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 08 16:34:27.518446 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:27.518446    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-tls podName:6ad20774-3ad0-4942-a467-66e8da1044fe nodeName:}" failed. No retries permitted until 2026-07-08 16:34:28.018424494 +0000 UTC m=+59.957025756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-4zbl2" (UID: "6ad20774-3ad0-4942-a467-66e8da1044fe") : secret "openshift-state-metrics-tls" not found
Jul 08 16:34:27.518643 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.518582    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ad20774-3ad0-4942-a467-66e8da1044fe-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.520862 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.520842    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.553642 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.553612    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lw2\" (UniqueName: \"kubernetes.io/projected/6ad20774-3ad0-4942-a467-66e8da1044fe-kube-api-access-p5lw2\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:27.619612 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvxw\" (UniqueName: \"kubernetes.io/projected/25608786-14b6-429f-912c-1ca46206e171-kube-api-access-jnvxw\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619612 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-node-exporter-wtmp\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619849 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25608786-14b6-429f-912c-1ca46206e171-node-exporter-accelerators-collector-config\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619849 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25608786-14b6-429f-912c-1ca46206e171-metrics-client-ca\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619849 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-root\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619849 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25608786-14b6-429f-912c-1ca46206e171-node-exporter-textfile\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619849 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25608786-14b6-429f-912c-1ca46206e171-node-exporter-tls\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.619849 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619819    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-node-exporter-wtmp\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620127 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619850    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-root\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620127 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25608786-14b6-429f-912c-1ca46206e171-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620127 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.619936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-sys\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620127 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.620011    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25608786-14b6-429f-912c-1ca46206e171-sys\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620127 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.620092    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/25608786-14b6-429f-912c-1ca46206e171-node-exporter-textfile\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620326 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.620244    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25608786-14b6-429f-912c-1ca46206e171-metrics-client-ca\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.620326 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.620295    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/25608786-14b6-429f-912c-1ca46206e171-node-exporter-accelerators-collector-config\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.622121 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.622100    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25608786-14b6-429f-912c-1ca46206e171-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.622234 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.622191    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/25608786-14b6-429f-912c-1ca46206e171-node-exporter-tls\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.630974 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.630945    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvxw\" (UniqueName: \"kubernetes.io/projected/25608786-14b6-429f-912c-1ca46206e171-kube-api-access-jnvxw\") pod \"node-exporter-ctcsm\" (UID: \"25608786-14b6-429f-912c-1ca46206e171\") " pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.773202 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:27.773163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ctcsm"
Jul 08 16:34:27.822696 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:27.822659    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25608786_14b6_429f_912c_1ca46206e171.slice/crio-146a90b44a6c08d7d5d29b1b60ca92bb045eb33cfe145db0167d61806db0deb6 WatchSource:0}: Error finding container 146a90b44a6c08d7d5d29b1b60ca92bb045eb33cfe145db0167d61806db0deb6: Status 404 returned error can't find the container with id 146a90b44a6c08d7d5d29b1b60ca92bb045eb33cfe145db0167d61806db0deb6
Jul 08 16:34:28.027289 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.025729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:28.027980 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.027958    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ad20774-3ad0-4942-a467-66e8da1044fe-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-4zbl2\" (UID: \"6ad20774-3ad0-4942-a467-66e8da1044fe\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:28.046206 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.046165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcsm" event={"ID":"25608786-14b6-429f-912c-1ca46206e171","Type":"ContainerStarted","Data":"146a90b44a6c08d7d5d29b1b60ca92bb045eb33cfe145db0167d61806db0deb6"}
Jul 08 16:34:28.048048 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.048022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j58g5" event={"ID":"fcc2ec40-2691-48c8-88df-a6f7e67ee07c","Type":"ContainerStarted","Data":"fe28db5892de1f0a0bf35f24017c203f2df5ef191bfbfaa67390cae877cbfc55"}
Jul 08 16:34:28.067435 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.067391    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j58g5" podStartSLOduration=1.920479321 podStartE2EDuration="11.067377961s" totalImagesPullingTime="9.14689864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:28.066741371 +0000 UTC m=+60.005342640" watchObservedRunningTime="2026-07-08 16:34:28.067377961 +0000 UTC m=+60.005979230"
Jul 08 16:34:28.317292 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.317201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"
Jul 08 16:34:28.494652 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.494611    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2"]
Jul 08 16:34:28.499611 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:28.499578    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad20774_3ad0_4942_a467_66e8da1044fe.slice/crio-444183fe530f9c037725b6793a282724cc3e4534c397bf0091b69cecea7e5966 WatchSource:0}: Error finding container 444183fe530f9c037725b6793a282724cc3e4534c397bf0091b69cecea7e5966: Status 404 returned error can't find the container with id 444183fe530f9c037725b6793a282724cc3e4534c397bf0091b69cecea7e5966
Jul 08 16:34:28.950161 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:28.950135    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6m2cv"
Jul 08 16:34:29.053827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:29.053796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2" event={"ID":"6ad20774-3ad0-4942-a467-66e8da1044fe","Type":"ContainerStarted","Data":"d21a2ab7c070eb7e86f8e63ef5b66d093fe12cf7052fe91cfa2df2c72265f2b4"}
Jul 08 16:34:29.053906 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:29.053854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2" event={"ID":"6ad20774-3ad0-4942-a467-66e8da1044fe","Type":"ContainerStarted","Data":"46ed51dc2dd7e84230a61e2c09ca43d5e67691a2d2cf375c4aafdbc1bdd50b1c"}
Jul 08 16:34:29.053906 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:29.053875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2" event={"ID":"6ad20774-3ad0-4942-a467-66e8da1044fe","Type":"ContainerStarted","Data":"444183fe530f9c037725b6793a282724cc3e4534c397bf0091b69cecea7e5966"}
Jul 08 16:34:30.059076 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.058978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2" event={"ID":"6ad20774-3ad0-4942-a467-66e8da1044fe","Type":"ContainerStarted","Data":"868cd77e27d0a52ee4ede8b3ff7ab6b77638601790c52affbb62196abf263695"}
Jul 08 16:34:30.060551 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.060531    2561 generic.go:356] "Generic (PLEG): container finished" podID="25608786-14b6-429f-912c-1ca46206e171" containerID="be0b7029ea147eb017a3529d45aec5de7b8bc501bb853ad0b849111c345eb4ba" exitCode=0
Jul 08 16:34:30.060645 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.060569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcsm" event={"ID":"25608786-14b6-429f-912c-1ca46206e171","Type":"ContainerDied","Data":"be0b7029ea147eb017a3529d45aec5de7b8bc501bb853ad0b849111c345eb4ba"}
Jul 08 16:34:30.077781 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.077731    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-4zbl2" podStartSLOduration=2.16572001 podStartE2EDuration="3.077712971s" totalImagesPullingTime="911.992961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:30.077494868 +0000 UTC m=+62.016096144" watchObservedRunningTime="2026-07-08 16:34:30.077712971 +0000 UTC m=+62.016314237"
Jul 08 16:34:30.530700 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.530646    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-97cbc85c4-m9smd"]
Jul 08 16:34:30.533612 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.533588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.536238 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536209    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 08 16:34:30.536604 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536587    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sl7vt\""
Jul 08 16:34:30.536712 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 08 16:34:30.536712 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536641    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 08 16:34:30.536712 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536669    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-98i49qhuhuooa\""
Jul 08 16:34:30.536712 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 08 16:34:30.536712 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.536706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 08 16:34:30.549268 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.549244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-97cbc85c4-m9smd"]
Jul 08 16:34:30.648895 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.648859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-tls\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.648895 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.648895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.649102 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.648914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.649102 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.648999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.649102 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.649077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a08708d-7963-47ce-8f34-45de1f6337c2-metrics-client-ca\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.649197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.649126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.649197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.649143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-grpc-tls\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.649197 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.649161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pvq\" (UniqueName: \"kubernetes.io/projected/0a08708d-7963-47ce-8f34-45de1f6337c2-kube-api-access-g5pvq\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750509 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750509 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-grpc-tls\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750743 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pvq\" (UniqueName: \"kubernetes.io/projected/0a08708d-7963-47ce-8f34-45de1f6337c2-kube-api-access-g5pvq\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750743 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-tls\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750828 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750828 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.750928 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.750825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.751058 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.751014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a08708d-7963-47ce-8f34-45de1f6337c2-metrics-client-ca\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.751811 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.751766    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a08708d-7963-47ce-8f34-45de1f6337c2-metrics-client-ca\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.753510 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.753483    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-grpc-tls\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.753701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.753659    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-tls\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.753767 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.753709    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.753767 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.753732    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.753767 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.753748    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.753856 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.753822    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0a08708d-7963-47ce-8f34-45de1f6337c2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.760520 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.760495    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pvq\" (UniqueName: \"kubernetes.io/projected/0a08708d-7963-47ce-8f34-45de1f6337c2-kube-api-access-g5pvq\") pod \"thanos-querier-97cbc85c4-m9smd\" (UID: \"0a08708d-7963-47ce-8f34-45de1f6337c2\") " pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.842825 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.842735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:30.976757 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:30.976632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-97cbc85c4-m9smd"]
Jul 08 16:34:30.980650 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:30.980620    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a08708d_7963_47ce_8f34_45de1f6337c2.slice/crio-6e2632db473ffbd2ecc530353a19bea4992d1b10523bd5d60e90e0a8a27a91d7 WatchSource:0}: Error finding container 6e2632db473ffbd2ecc530353a19bea4992d1b10523bd5d60e90e0a8a27a91d7: Status 404 returned error can't find the container with id 6e2632db473ffbd2ecc530353a19bea4992d1b10523bd5d60e90e0a8a27a91d7
Jul 08 16:34:31.064763 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:31.064730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"6e2632db473ffbd2ecc530353a19bea4992d1b10523bd5d60e90e0a8a27a91d7"}
Jul 08 16:34:31.066867 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:31.066837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcsm" event={"ID":"25608786-14b6-429f-912c-1ca46206e171","Type":"ContainerStarted","Data":"8db88e977b6ba5f5934a2f01b052e5e2f751de4254dcc4931feab5a2de77b9c3"}
Jul 08 16:34:31.066978 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:31.066888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcsm" event={"ID":"25608786-14b6-429f-912c-1ca46206e171","Type":"ContainerStarted","Data":"c113b97adf5c7bcc6de512645448cec6ba2cc11aa04054c7ceb396acf3a498f6"}
Jul 08 16:34:31.086283 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:31.086147    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ctcsm" podStartSLOduration=2.794270815 podStartE2EDuration="4.086129574s" totalImagesPullingTime="1.228326318s" totalInitContainerRuntime="63.532441ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:31.085847857 +0000 UTC m=+63.024449127" watchObservedRunningTime="2026-07-08 16:34:31.086129574 +0000 UTC m=+63.024730822"
Jul 08 16:34:33.076567 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.076489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"5ec31f40f3171ab4781ab96d8b52a6ad538e0470e899006208f749709abdf6b1"}
Jul 08 16:34:33.076567 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.076549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"bbdf7343607cccbeeeeebda79fde61aac601fcf6a9c301df605a4e9034f536ca"}
Jul 08 16:34:33.076965 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.076576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"2d128800069d25df5da0a77c1b00a37df35a6ab6a9a61912a7cfd48a0e99b941"}
Jul 08 16:34:33.613921 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.613890    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:34:33.616975 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.616948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.619729 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.619706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 16:34:33.619852 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.619751    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 16:34:33.619945 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.619873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2svfvtnjj5mut\""
Jul 08 16:34:33.620007 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.619970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 16:34:33.620967 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.620936    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 16:34:33.621082 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.620945    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 16:34:33.621082 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621048    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 16:34:33.621600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621462    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 16:34:33.621600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621476    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nd25h\""
Jul 08 16:34:33.621600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621505    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 16:34:33.621600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621516    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 16:34:33.621600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621603    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 16:34:33.621902 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621884    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 16:34:33.621961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621884    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 16:34:33.621961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.621905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 16:34:33.622663 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.622507    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 16:34:33.625287 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.625270    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 16:34:33.640961 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.640931    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzztg\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-kube-api-access-zzztg\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-config\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.676713 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.677628 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.675993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.677628 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.676023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.677628 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.676058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.677628 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.676083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.776503 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.776503 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzztg\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-kube-api-access-zzztg\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-config\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.776995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.777022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.777641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.777052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.778472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.777086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.778472 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:34:33.777784    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle podName:b1fb2167-28cf-4a1a-997d-f2347b629b31 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:34.277762528 +0000 UTC m=+66.216363779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle") pod "prometheus-k8s-0" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31") : configmap references non-existent config key: ca-bundle.crt
Jul 08 16:34:33.778472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.778096    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.779506 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.778902    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.779506 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.779150    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.781855 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.781542    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.784304 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.784157    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.784391 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.784356    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.784620 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.784599    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.784865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.784839    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.785988 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.785939    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.786141 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.786118    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.786568 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.786549    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.787308 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.787094    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.787308 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.787246    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.787308 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.787258    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.787523 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.786571    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.788607 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.788553    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.788753 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.788728    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.789284 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.789263    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-config\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:33.790338 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:33.790318    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzztg\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-kube-api-access-zzztg\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:34.281295 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.281268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:34.282196 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.282172    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:34.482772 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.482736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:34.485654 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.485635    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:34:34.496216 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.496201    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:34:34.506699 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.506670    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vts45\" (UniqueName: \"kubernetes.io/projected/9a6f26ab-7d25-4a7c-b527-d7a717586e35-kube-api-access-vts45\") pod \"network-check-target-ddnsp\" (UID: \"9a6f26ab-7d25-4a7c-b527-d7a717586e35\") " pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:34.528653 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.528631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:34:34.583862 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.583833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:34.584003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.583915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:34.586571 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.586553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:34:34.586700 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.586668    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:34:34.597567 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.597536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b11058bb-8eed-4973-b455-a320ededc11b-metrics-certs\") pod \"network-metrics-daemon-pdp4m\" (UID: \"b11058bb-8eed-4973-b455-a320ededc11b\") " pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:34.597659 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.597614    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e71b89-45d3-415c-844d-c81aa15d87f7-original-pull-secret\") pod \"global-pull-secret-syncer-dlb5z\" (UID: \"03e71b89-45d3-415c-844d-c81aa15d87f7\") " pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:34.680637 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.680605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:34:34.684003 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:34.683977    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fb2167_28cf_4a1a_997d_f2347b629b31.slice/crio-99cffaafb8b58cffbe1a4acc1ced60f9b7c21f6485578fd18a6a7bfcf8d1f8bf WatchSource:0}: Error finding container 99cffaafb8b58cffbe1a4acc1ced60f9b7c21f6485578fd18a6a7bfcf8d1f8bf: Status 404 returned error can't find the container with id 99cffaafb8b58cffbe1a4acc1ced60f9b7c21f6485578fd18a6a7bfcf8d1f8bf
Jul 08 16:34:34.717448 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.717425    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s7kjq\""
Jul 08 16:34:34.725051 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.725031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:34.726926 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.726907    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4cx7\""
Jul 08 16:34:34.731247 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.731228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dlb5z"
Jul 08 16:34:34.734112 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.734092    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdp4m"
Jul 08 16:34:34.865227 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:34.865198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ddnsp"]
Jul 08 16:34:34.868670 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:34.868638    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6f26ab_7d25_4a7c_b527_d7a717586e35.slice/crio-cb1696158a8486d8553f13137ef7222615058197842664127f7cb00275a9f40c WatchSource:0}: Error finding container cb1696158a8486d8553f13137ef7222615058197842664127f7cb00275a9f40c: Status 404 returned error can't find the container with id cb1696158a8486d8553f13137ef7222615058197842664127f7cb00275a9f40c
Jul 08 16:34:35.041224 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.041137    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-czxrs"
Jul 08 16:34:35.085745 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.085714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddnsp" event={"ID":"9a6f26ab-7d25-4a7c-b527-d7a717586e35","Type":"ContainerStarted","Data":"cb1696158a8486d8553f13137ef7222615058197842664127f7cb00275a9f40c"}
Jul 08 16:34:35.089278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.089214    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdp4m"]
Jul 08 16:34:35.090450 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.090086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"9e81c50fedeae0ab05f51113ae72fbf0771e903be0f2864f93135a2ef0165f60"}
Jul 08 16:34:35.090450 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.090171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"22a75214a8142d20c710ae766afdd1dcd14c2d3e53edf95435e1c77fce75f3f4"}
Jul 08 16:34:35.090450 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.090304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:35.090450 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.090380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" event={"ID":"0a08708d-7963-47ce-8f34-45de1f6337c2","Type":"ContainerStarted","Data":"a00b3db39b152a714c6a0ae95eb22604791429389a6388d2341cd188c620631b"}
Jul 08 16:34:35.092486 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.092466    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dlb5z"]
Jul 08 16:34:35.092766 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.092740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"99cffaafb8b58cffbe1a4acc1ced60f9b7c21f6485578fd18a6a7bfcf8d1f8bf"}
Jul 08 16:34:35.093833 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:35.093806    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11058bb_8eed_4973_b455_a320ededc11b.slice/crio-335b2b992943e33cc85be70c586a687fc206cde89e9a2f23cfc5e362e5d39e06 WatchSource:0}: Error finding container 335b2b992943e33cc85be70c586a687fc206cde89e9a2f23cfc5e362e5d39e06: Status 404 returned error can't find the container with id 335b2b992943e33cc85be70c586a687fc206cde89e9a2f23cfc5e362e5d39e06
Jul 08 16:34:35.097489 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:34:35.097453    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e71b89_45d3_415c_844d_c81aa15d87f7.slice/crio-de908bd8b150ab6917609330d8d60254eafcbb0662a18dee5b22545054fdc177 WatchSource:0}: Error finding container de908bd8b150ab6917609330d8d60254eafcbb0662a18dee5b22545054fdc177: Status 404 returned error can't find the container with id de908bd8b150ab6917609330d8d60254eafcbb0662a18dee5b22545054fdc177
Jul 08 16:34:35.115859 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:35.115820    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd" podStartSLOduration=2.130097679 podStartE2EDuration="5.115799557s" totalImagesPullingTime="2.985701878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:35.114624156 +0000 UTC m=+67.053225452" watchObservedRunningTime="2026-07-08 16:34:35.115799557 +0000 UTC m=+67.054400872"
Jul 08 16:34:36.104311 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:36.104274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dlb5z" event={"ID":"03e71b89-45d3-415c-844d-c81aa15d87f7","Type":"ContainerStarted","Data":"de908bd8b150ab6917609330d8d60254eafcbb0662a18dee5b22545054fdc177"}
Jul 08 16:34:36.105925 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:36.105888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdp4m" event={"ID":"b11058bb-8eed-4973-b455-a320ededc11b","Type":"ContainerStarted","Data":"335b2b992943e33cc85be70c586a687fc206cde89e9a2f23cfc5e362e5d39e06"}
Jul 08 16:34:37.118561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:37.117664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdp4m" event={"ID":"b11058bb-8eed-4973-b455-a320ededc11b","Type":"ContainerStarted","Data":"00d0f1b3a8658bd5a9f5a870f28e290bcd9ff640a6dc72b6ed80aa363262135d"}
Jul 08 16:34:37.118561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:37.117748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdp4m" event={"ID":"b11058bb-8eed-4973-b455-a320ededc11b","Type":"ContainerStarted","Data":"730d518a5d93c7dfb42cfc6440119121de6e08b879780e30a16052ca5a74d048"}
Jul 08 16:34:37.120838 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:37.120807    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7" exitCode=0
Jul 08 16:34:37.120977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:37.120855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"}
Jul 08 16:34:37.137222 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:37.137170    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pdp4m" podStartSLOduration=66.853262668 podStartE2EDuration="1m8.137153518s" totalImagesPullingTime="1.28389085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:37.135117403 +0000 UTC m=+69.073718667" watchObservedRunningTime="2026-07-08 16:34:37.137153518 +0000 UTC m=+69.075754787"
Jul 08 16:34:41.113202 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:41.113170    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-97cbc85c4-m9smd"
Jul 08 16:34:42.153444 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.153407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"}
Jul 08 16:34:42.153764 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.153569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"}
Jul 08 16:34:42.156329 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.155390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dlb5z" event={"ID":"03e71b89-45d3-415c-844d-c81aa15d87f7","Type":"ContainerStarted","Data":"a5ef5f64c793e55eb8344272fad1e0f04b37fa10348c8ad2646200f4311300a4"}
Jul 08 16:34:42.158067 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.157959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ddnsp" event={"ID":"9a6f26ab-7d25-4a7c-b527-d7a717586e35","Type":"ContainerStarted","Data":"b71e214b7b61fa1412c7b50f3c98eec7c3eb2935c7e1bad22c25ca0595e4ebc1"}
Jul 08 16:34:42.158249 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.158232    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:34:42.195060 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.194963    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ddnsp" podStartSLOduration=66.109291356 podStartE2EDuration="1m13.194938797s" totalImagesPullingTime="7.085647441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:42.193758009 +0000 UTC m=+74.132359280" watchObservedRunningTime="2026-07-08 16:34:42.194938797 +0000 UTC m=+74.133540058"
Jul 08 16:34:42.196608 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:42.196557    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dlb5z" podStartSLOduration=66.323963466 podStartE2EDuration="1m13.19653991s" totalImagesPullingTime="6.872576444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:42.173069236 +0000 UTC m=+74.111670506" watchObservedRunningTime="2026-07-08 16:34:42.19653991 +0000 UTC m=+74.135141183"
Jul 08 16:34:43.165014 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:43.164973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"}
Jul 08 16:34:43.165476 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:43.165046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"}
Jul 08 16:34:43.165476 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:43.165080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"}
Jul 08 16:34:43.165476 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:43.165105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerStarted","Data":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"}
Jul 08 16:34:43.202016 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:43.201962    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.636122279 podStartE2EDuration="10.201944476s" totalImagesPullingTime="6.537370589s" totalInitContainerRuntime="28.451608ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:43.199858543 +0000 UTC m=+75.138459824" watchObservedRunningTime="2026-07-08 16:34:43.201944476 +0000 UTC m=+75.140545746"
Jul 08 16:34:44.529508 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:34:44.529474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:13.168041 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:13.167908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ddnsp"
Jul 08 16:35:34.529129 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:34.529087    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:34.548369 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:34.548342    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:35.362601 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:35.362574    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.089956 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.089919    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:35:52.090463 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.090315    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="prometheus" containerID="cri-o://dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120" gracePeriod=600
Jul 08 16:35:52.090463 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.090340    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-web" containerID="cri-o://6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4" gracePeriod=600
Jul 08 16:35:52.090463 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.090403    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy" containerID="cri-o://6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05" gracePeriod=600
Jul 08 16:35:52.090623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.090342    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72" gracePeriod=600
Jul 08 16:35:52.090623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.090416    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="config-reloader" containerID="cri-o://3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f" gracePeriod=600
Jul 08 16:35:52.090756 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.090372    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="thanos-sidecar" containerID="cri-o://e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b" gracePeriod=600
Jul 08 16:35:52.335082 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.335055    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.416993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.416950    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72" exitCode=0
Jul 08 16:35:52.416993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.416983    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05" exitCode=0
Jul 08 16:35:52.416993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.416992    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4" exitCode=0
Jul 08 16:35:52.416993 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417000    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b" exitCode=0
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417009    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f" exitCode=0
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417018    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120" exitCode=0
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"}
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"}
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417191    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"}
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"}
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417250    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"}
Jul 08 16:35:52.417348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"}
Jul 08 16:35:52.417888 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.417372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1fb2167-28cf-4a1a-997d-f2347b629b31","Type":"ContainerDied","Data":"99cffaafb8b58cffbe1a4acc1ced60f9b7c21f6485578fd18a6a7bfcf8d1f8bf"}
Jul 08 16:35:52.426635 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.426613    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.434319 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.434300    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.440959 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.440939    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.447482 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.447453    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.453898 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.453880    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.460876 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.460847    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.467294 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.467280    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.467557 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.467532    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.467607 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.467559    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"} err="failed to get container status \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist"
Jul 08 16:35:52.467607 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.467590    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.467848 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.467831    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.467906 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.467853    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"} err="failed to get container status \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist"
Jul 08 16:35:52.467906 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.467869    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.468101 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.468086    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.468141 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468107    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"} err="failed to get container status \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist"
Jul 08 16:35:52.468141 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468121    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.468380 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.468353    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.468430 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468386    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"} err="failed to get container status \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist"
Jul 08 16:35:52.468430 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468407    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.468652 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.468637    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.468724 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468664    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"} err="failed to get container status \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist"
Jul 08 16:35:52.468724 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468676    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.468979 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.468961    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.469030 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.468987    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"} err="failed to get container status \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist"
Jul 08 16:35:52.469030 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469004    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.469233 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:35:52.469216    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.469278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469237    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"} err="failed to get container status \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist"
Jul 08 16:35:52.469278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469249    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.469484 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469469    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"} err="failed to get container status \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist"
Jul 08 16:35:52.469484 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469483    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.469811 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469797    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"} err="failed to get container status \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist"
Jul 08 16:35:52.469867 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.469811    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.470025 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470010    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"} err="failed to get container status \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist"
Jul 08 16:35:52.470025 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470025    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.470266 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470249    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"} err="failed to get container status \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist"
Jul 08 16:35:52.470304 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470265    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.470520 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470501    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"} err="failed to get container status \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist"
Jul 08 16:35:52.470591 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470524    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.470790 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470771    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"} err="failed to get container status \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist"
Jul 08 16:35:52.470850 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470790    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.470990 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470975    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"} err="failed to get container status \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist"
Jul 08 16:35:52.471036 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.470990    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.471189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471171    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"} err="failed to get container status \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist"
Jul 08 16:35:52.471236 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471190    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.471375 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471355    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"} err="failed to get container status \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist"
Jul 08 16:35:52.471417 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471376    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.471590 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471572    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"} err="failed to get container status \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist"
Jul 08 16:35:52.471663 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471591    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.471810 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471793    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"} err="failed to get container status \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist"
Jul 08 16:35:52.471859 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471810    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.471977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471964    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"} err="failed to get container status \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist"
Jul 08 16:35:52.472020 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.471977    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.472160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472144    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"} err="failed to get container status \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist"
Jul 08 16:35:52.472210 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472163    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.472397 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472380    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"} err="failed to get container status \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist"
Jul 08 16:35:52.472467 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472399    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.472609 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472595    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"} err="failed to get container status \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist"
Jul 08 16:35:52.472651 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472608    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.472920 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472899    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"} err="failed to get container status \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist"
Jul 08 16:35:52.472976 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.472921    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.473136 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473120    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"} err="failed to get container status \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist"
Jul 08 16:35:52.473180 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473137    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.473340 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473323    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"} err="failed to get container status \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist"
Jul 08 16:35:52.473379 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473341    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.473531 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473517    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"} err="failed to get container status \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist"
Jul 08 16:35:52.473581 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473531    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.473755 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473732    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"} err="failed to get container status \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist"
Jul 08 16:35:52.473801 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473757    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.473955 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473941    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"} err="failed to get container status \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist"
Jul 08 16:35:52.473997 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.473955    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.474154 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474139    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"} err="failed to get container status \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist"
Jul 08 16:35:52.474204 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474154    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.474358 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474340    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"} err="failed to get container status \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist"
Jul 08 16:35:52.474400 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474359    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.474528 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474514    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"} err="failed to get container status \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist"
Jul 08 16:35:52.474580 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474528    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.474750 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474732    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"} err="failed to get container status \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist"
Jul 08 16:35:52.474827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474752    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.474959 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474944    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"} err="failed to get container status \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist"
Jul 08 16:35:52.475003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.474959    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.475151 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475135    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"} err="failed to get container status \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist"
Jul 08 16:35:52.475218 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475153    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.475342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475328    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"} err="failed to get container status \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist"
Jul 08 16:35:52.475387 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475342    2561 scope.go:117] "RemoveContainer" containerID="0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"
Jul 08 16:35:52.475515 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475500    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72"} err="failed to get container status \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": rpc error: code = NotFound desc = could not find container \"0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72\": container with ID starting with 0603f3dbc962cb9266f6babbefb1b6b2b12df262f6412d3f415e85ab4527ac72 not found: ID does not exist"
Jul 08 16:35:52.475578 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475517    2561 scope.go:117] "RemoveContainer" containerID="6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"
Jul 08 16:35:52.475723 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475705    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05"} err="failed to get container status \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": rpc error: code = NotFound desc = could not find container \"6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05\": container with ID starting with 6d671bc357d92454280e0d51049f50383ad05ca6c8592ce7396b2712cf541b05 not found: ID does not exist"
Jul 08 16:35:52.475771 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475723    2561 scope.go:117] "RemoveContainer" containerID="6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"
Jul 08 16:35:52.475905 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475889    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4"} err="failed to get container status \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": rpc error: code = NotFound desc = could not find container \"6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4\": container with ID starting with 6d132bde85da44eaa54f3f498756d215ca612a2495cdeeb820093b7d505112d4 not found: ID does not exist"
Jul 08 16:35:52.475947 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.475906    2561 scope.go:117] "RemoveContainer" containerID="e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"
Jul 08 16:35:52.476084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476070    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b"} err="failed to get container status \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": rpc error: code = NotFound desc = could not find container \"e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b\": container with ID starting with e4059829241151d3ec9907007972faef44af7841a8dc1f1396da1fb208295e6b not found: ID does not exist"
Jul 08 16:35:52.476084 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476084    2561 scope.go:117] "RemoveContainer" containerID="3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"
Jul 08 16:35:52.476260 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476244    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f"} err="failed to get container status \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": rpc error: code = NotFound desc = could not find container \"3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f\": container with ID starting with 3cfb0ac1e9b8ba26e724fdd085c4c75fa8ceebe557e4da6ddc7c531b4ce0146f not found: ID does not exist"
Jul 08 16:35:52.476319 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476262    2561 scope.go:117] "RemoveContainer" containerID="dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"
Jul 08 16:35:52.476449 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476430    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120"} err="failed to get container status \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": rpc error: code = NotFound desc = could not find container \"dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120\": container with ID starting with dfd235b967cf7c7c17bcfd3e37543d27e876d597d714c16fb83eb91a2310a120 not found: ID does not exist"
Jul 08 16:35:52.476489 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476452    2561 scope.go:117] "RemoveContainer" containerID="27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"
Jul 08 16:35:52.476630 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.476613    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7"} err="failed to get container status \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": rpc error: code = NotFound desc = could not find container \"27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7\": container with ID starting with 27f92d6200c259db97a8b33e7c41e6d4f483a72a07e77dc1bc6a30b0691423c7 not found: ID does not exist"
Jul 08 16:35:52.527902 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.527871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.527917    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzztg\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-kube-api-access-zzztg\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.527945    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-2\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.527970    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-kube-rbac-proxy\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528254 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528094    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-tls-assets\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528254 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528159    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-db\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528254 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528227    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-web-config\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528406 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-metrics-client-certs\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528406 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-grpc-tls\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528406 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528317    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-metrics-client-ca\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528406 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528342    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528406 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528374    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-1\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528406 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528397    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-config\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-tls\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528378    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-kubelet-serving-ca-bundle\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-serving-certs-ca-bundle\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-thanos-prometheus-http-client-file\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528541    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-0\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.528701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528621    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-config-out\") pod \"b1fb2167-28cf-4a1a-997d-f2347b629b31\" (UID: \"b1fb2167-28cf-4a1a-997d-f2347b629b31\") "
Jul 08 16:35:52.529478 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528746    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.529478 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528883    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-metrics-client-ca\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.529478 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.528902    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.529478 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.529280    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.529705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.529553    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.529705 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.529629    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:35:52.530445 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.530411    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.530885 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.530862    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.531045 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.531018    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:52.531212 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.531152    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:52.531212 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.531162    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.531585 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.531539    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.531585 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.531551    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-kube-api-access-zzztg" (OuterVolumeSpecName: "kube-api-access-zzztg") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "kube-api-access-zzztg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:52.532002 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.531976    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.532262 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.532235    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.533374 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.533346    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-config-out" (OuterVolumeSpecName: "config-out") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:35:52.533374 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.533346    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.533492 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.533418    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.533638 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.533621    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.533743 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.533722    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-config" (OuterVolumeSpecName: "config") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.542229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.542208    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-web-config" (OuterVolumeSpecName: "web-config") pod "b1fb2167-28cf-4a1a-997d-f2347b629b31" (UID: "b1fb2167-28cf-4a1a-997d-f2347b629b31"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:52.629354 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629320    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629354 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629349    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629354 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629361    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629371    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629380    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629393    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629406    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629417    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629428    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-config-out\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629436    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629444    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzztg\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-kube-api-access-zzztg\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629452    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-kube-rbac-proxy\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629460    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1fb2167-28cf-4a1a-997d-f2347b629b31-tls-assets\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629467    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-k8s-db\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629476    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-web-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629484    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-metrics-client-certs\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629493    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1fb2167-28cf-4a1a-997d-f2347b629b31-secret-grpc-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.629577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.629501    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1fb2167-28cf-4a1a-997d-f2347b629b31-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:35:52.743266 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.743233    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:35:52.748866 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.748840    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:35:52.782725 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782697    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:35:52.782951 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782939    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="init-config-reloader"
Jul 08 16:35:52.782992 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="init-config-reloader"
Jul 08 16:35:52.782992 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782962    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-web"
Jul 08 16:35:52.782992 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782968    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-web"
Jul 08 16:35:52.782992 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782975    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy"
Jul 08 16:35:52.782992 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782980    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy"
Jul 08 16:35:52.782992 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782990    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="thanos-sidecar"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.782995    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="thanos-sidecar"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783001    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-thanos"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783007    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-thanos"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783015    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="prometheus"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783019    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="prometheus"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783028    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="config-reloader"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783033    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="config-reloader"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783074    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="thanos-sidecar"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783081    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-thanos"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783087    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy-web"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783094    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="kube-rbac-proxy"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783098    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="prometheus"
Jul 08 16:35:52.783160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.783104    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" containerName="config-reloader"
Jul 08 16:35:52.786906 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.786886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.789913 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.789880    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 16:35:52.790033 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.789914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 16:35:52.790033 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 16:35:52.790033 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790027    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 16:35:52.790206 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790036    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 16:35:52.790206 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nd25h\""
Jul 08 16:35:52.790281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 16:35:52.790420 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790403    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 16:35:52.790471 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 16:35:52.790536 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790521    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 16:35:52.790613 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790596    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 16:35:52.790701 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790602    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 16:35:52.790754 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790739    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 16:35:52.790806 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790751    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2svfvtnjj5mut\""
Jul 08 16:35:52.790806 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.790758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 16:35:52.794038 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.793899    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 16:35:52.796730 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.796713    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 16:35:52.800834 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.800815    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1fb2167-28cf-4a1a-997d-f2347b629b31" path="/var/lib/kubelet/pods/b1fb2167-28cf-4a1a-997d-f2347b629b31/volumes"
Jul 08 16:35:52.806183 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.806157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:35:52.931977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.931935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.931977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.931974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.931995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2429eaff-a852-4855-9508-1a83644fa586-config-out\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-web-config\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdgm\" (UniqueName: \"kubernetes.io/projected/2429eaff-a852-4855-9508-1a83644fa586-kube-api-access-qwdgm\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2429eaff-a852-4855-9508-1a83644fa586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932472 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-config\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:52.932760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:52.932583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033337 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2429eaff-a852-4855-9508-1a83644fa586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033337 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033337 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033337 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-config\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033623 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.033983 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.033958    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034199 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034160    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2429eaff-a852-4855-9508-1a83644fa586-config-out\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034464 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034464 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034464 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-web-config\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034464 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034484    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdgm\" (UniqueName: \"kubernetes.io/projected/2429eaff-a852-4855-9508-1a83644fa586-kube-api-access-qwdgm\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.034657 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.034598    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.035352 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.035022    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.036702 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.036127    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.036702 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.036635    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.036864 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.036839    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.037041 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.037020    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2429eaff-a852-4855-9508-1a83644fa586-config-out\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.037484 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.037463    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2429eaff-a852-4855-9508-1a83644fa586-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.037897 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.037846    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2429eaff-a852-4855-9508-1a83644fa586-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.038444 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.038245    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.038444 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.038342    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.038565 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.038500    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-config\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.039150 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.038708    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.039150 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.038714    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.039150 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.039038    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.039538 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.039507    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.039834 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.039817    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2429eaff-a852-4855-9508-1a83644fa586-web-config\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.044956 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.044932    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdgm\" (UniqueName: \"kubernetes.io/projected/2429eaff-a852-4855-9508-1a83644fa586-kube-api-access-qwdgm\") pod \"prometheus-k8s-0\" (UID: \"2429eaff-a852-4855-9508-1a83644fa586\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.097067 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.097025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:35:53.246854 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.246813    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 16:35:53.252141 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:35:53.252116    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2429eaff_a852_4855_9508_1a83644fa586.slice/crio-4b5b4642e61e18c137d919adbd0a843f34dc121f5b034f404fb72e6ccce57f4c WatchSource:0}: Error finding container 4b5b4642e61e18c137d919adbd0a843f34dc121f5b034f404fb72e6ccce57f4c: Status 404 returned error can't find the container with id 4b5b4642e61e18c137d919adbd0a843f34dc121f5b034f404fb72e6ccce57f4c
Jul 08 16:35:53.422186 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.422143    2561 generic.go:356] "Generic (PLEG): container finished" podID="2429eaff-a852-4855-9508-1a83644fa586" containerID="4d3dcf0c18be6c9fabe0fcf1be4797c022e8b7ff5a9647143941a0e9b18513c3" exitCode=0
Jul 08 16:35:53.422346 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.422224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerDied","Data":"4d3dcf0c18be6c9fabe0fcf1be4797c022e8b7ff5a9647143941a0e9b18513c3"}
Jul 08 16:35:53.422346 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:53.422332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"4b5b4642e61e18c137d919adbd0a843f34dc121f5b034f404fb72e6ccce57f4c"}
Jul 08 16:35:54.430379 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.430345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"5735fb889719a462d82363ce2bedc0d88e0d466422763a34f98771eb19142cbb"}
Jul 08 16:35:54.430870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.430407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"f9a705d5c48cfda0ba1e974ee0aab5d320d823f12e0d4401a3caf696b2790dcd"}
Jul 08 16:35:54.430870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.430434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"afdff13b3803445a4515f463c7aa0cee1428f3d7f99e7e2a769312f380ab18c3"}
Jul 08 16:35:54.430870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.430458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"3be0f9cd4f8213acdc137069d20ff9a899919eb5f7791f2048df4480a7cc1371"}
Jul 08 16:35:54.430870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.430483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"5a62f39c2fedbe70552bb44e16a26e53f7fef039b2582daa7570b158c5bb7c6c"}
Jul 08 16:35:54.430870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.430506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2429eaff-a852-4855-9508-1a83644fa586","Type":"ContainerStarted","Data":"512f3bfba5786fbd255ce76d1fc1b36588e9cca23116dc1d86d705be9620e71e"}
Jul 08 16:35:54.465930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:54.465871    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.442630608 podStartE2EDuration="2.465849882s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.219274ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:54.464297336 +0000 UTC m=+146.402898640" watchObservedRunningTime="2026-07-08 16:35:54.465849882 +0000 UTC m=+146.404451150"
Jul 08 16:35:58.097822 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:35:58.097773    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:48.003596 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.003561    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"]
Jul 08 16:36:48.006765 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.006742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.010519 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.010488    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 16:36:48.010622 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.010527    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 16:36:48.011929 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.011896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pjvxp\""
Jul 08 16:36:48.012068 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.011960    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 08 16:36:48.012068 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.011990    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 16:36:48.024059 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.024039    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"]
Jul 08 16:36:48.172253 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.171417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ckh\" (UniqueName: \"kubernetes.io/projected/e1268083-93ba-4d05-86bb-0752b44d53f7-kube-api-access-65ckh\") pod \"managed-serviceaccount-addon-agent-578577b58b-wdjnp\" (UID: \"e1268083-93ba-4d05-86bb-0752b44d53f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.172253 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.171908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1268083-93ba-4d05-86bb-0752b44d53f7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578577b58b-wdjnp\" (UID: \"e1268083-93ba-4d05-86bb-0752b44d53f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.272802 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.272713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ckh\" (UniqueName: \"kubernetes.io/projected/e1268083-93ba-4d05-86bb-0752b44d53f7-kube-api-access-65ckh\") pod \"managed-serviceaccount-addon-agent-578577b58b-wdjnp\" (UID: \"e1268083-93ba-4d05-86bb-0752b44d53f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.272802 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.272774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1268083-93ba-4d05-86bb-0752b44d53f7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578577b58b-wdjnp\" (UID: \"e1268083-93ba-4d05-86bb-0752b44d53f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.275237 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.275210    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e1268083-93ba-4d05-86bb-0752b44d53f7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-578577b58b-wdjnp\" (UID: \"e1268083-93ba-4d05-86bb-0752b44d53f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.290415 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.290389    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ckh\" (UniqueName: \"kubernetes.io/projected/e1268083-93ba-4d05-86bb-0752b44d53f7-kube-api-access-65ckh\") pod \"managed-serviceaccount-addon-agent-578577b58b-wdjnp\" (UID: \"e1268083-93ba-4d05-86bb-0752b44d53f7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.323108 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.323080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"
Jul 08 16:36:48.448877 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.448836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp"]
Jul 08 16:36:48.452864 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:36:48.452835    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1268083_93ba_4d05_86bb_0752b44d53f7.slice/crio-ec14b1e30f9c1fe9f580e250b2e703052a13caf3ab0929676de29ff66c960da2 WatchSource:0}: Error finding container ec14b1e30f9c1fe9f580e250b2e703052a13caf3ab0929676de29ff66c960da2: Status 404 returned error can't find the container with id ec14b1e30f9c1fe9f580e250b2e703052a13caf3ab0929676de29ff66c960da2
Jul 08 16:36:48.627633 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:48.627553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp" event={"ID":"e1268083-93ba-4d05-86bb-0752b44d53f7","Type":"ContainerStarted","Data":"ec14b1e30f9c1fe9f580e250b2e703052a13caf3ab0929676de29ff66c960da2"}
Jul 08 16:36:51.640280 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:51.640239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp" event={"ID":"e1268083-93ba-4d05-86bb-0752b44d53f7","Type":"ContainerStarted","Data":"3f26d8cd6e6099cc4d1d13dd7be6660a876b0cdc283ef83eae6faf9c02e031f9"}
Jul 08 16:36:51.659120 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:51.659067    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-578577b58b-wdjnp" podStartSLOduration=2.085801109 podStartE2EDuration="4.659048573s" totalImagesPullingTime="2.573247464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:51.657861346 +0000 UTC m=+203.596462615" watchObservedRunningTime="2026-07-08 16:36:51.659048573 +0000 UTC m=+203.597649843"
Jul 08 16:36:53.098189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:53.098152    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:53.113116 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:53.113094    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:36:53.662923 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:36:53.662894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 16:38:28.670203 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:28.670175    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:38:41.019300 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.019263    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-pmfdj"]
Jul 08 16:38:41.022353 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.022336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.025915 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.025892    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 08 16:38:41.026052 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.025922    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 08 16:38:41.027309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.027271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 08 16:38:41.027309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.027297    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 08 16:38:41.027468 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.027334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lvjxc\""
Jul 08 16:38:41.037692 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.037647    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-pmfdj"]
Jul 08 16:38:41.125792 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.125756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb-certificates\") pod \"keda-admission-5fbd5c4644-pmfdj\" (UID: \"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb\") " pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.125792 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.125794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdq5m\" (UniqueName: \"kubernetes.io/projected/4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb-kube-api-access-vdq5m\") pod \"keda-admission-5fbd5c4644-pmfdj\" (UID: \"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb\") " pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.227229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.227196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb-certificates\") pod \"keda-admission-5fbd5c4644-pmfdj\" (UID: \"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb\") " pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.227229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.227232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdq5m\" (UniqueName: \"kubernetes.io/projected/4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb-kube-api-access-vdq5m\") pod \"keda-admission-5fbd5c4644-pmfdj\" (UID: \"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb\") " pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.230096 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.230064    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb-certificates\") pod \"keda-admission-5fbd5c4644-pmfdj\" (UID: \"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb\") " pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.236533 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.236502    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdq5m\" (UniqueName: \"kubernetes.io/projected/4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb-kube-api-access-vdq5m\") pod \"keda-admission-5fbd5c4644-pmfdj\" (UID: \"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb\") " pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.336192 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.336083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:41.487237 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.487182    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-pmfdj"]
Jul 08 16:38:41.489661 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:38:41.489631    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce039a4_fc4c_42ef_9ad3_62f7ddb6d4cb.slice/crio-0fc5bb5a4472a5b2519551401d0bbd1c1bd3f7f5d67c8463d70eab2e279ef73c WatchSource:0}: Error finding container 0fc5bb5a4472a5b2519551401d0bbd1c1bd3f7f5d67c8463d70eab2e279ef73c: Status 404 returned error can't find the container with id 0fc5bb5a4472a5b2519551401d0bbd1c1bd3f7f5d67c8463d70eab2e279ef73c
Jul 08 16:38:41.492471 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:41.491023    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:38:42.030218 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:42.030180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj" event={"ID":"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb","Type":"ContainerStarted","Data":"0fc5bb5a4472a5b2519551401d0bbd1c1bd3f7f5d67c8463d70eab2e279ef73c"}
Jul 08 16:38:46.046116 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:46.046073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj" event={"ID":"4ce039a4-fc4c-42ef-9ad3-62f7ddb6d4cb","Type":"ContainerStarted","Data":"cd20a4290e30f138229cc293d4774e257b1d2802274f49b12c5b8b368f74b0e6"}
Jul 08 16:38:46.046562 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:46.046228    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:38:46.067560 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:38:46.067506    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj" podStartSLOduration=1.214535844 podStartE2EDuration="5.067489858s" totalImagesPullingTime="3.852954014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:46.065119823 +0000 UTC m=+318.003721117" watchObservedRunningTime="2026-07-08 16:38:46.067489858 +0000 UTC m=+318.006091127"
Jul 08 16:39:07.052251 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:07.052217    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-pmfdj"
Jul 08 16:39:49.386633 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.386595    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-wfh2z"]
Jul 08 16:39:49.389547 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.389526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.393401 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.393370    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:39:49.394624 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.394600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-n29cq\""
Jul 08 16:39:49.395023 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.395007    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 16:39:49.395309 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.395031    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:39:49.402553 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.402534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wfh2z"]
Jul 08 16:39:49.441798 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.441775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjb6\" (UniqueName: \"kubernetes.io/projected/a9fe57fc-d963-420a-8ceb-25ed47e43a15-kube-api-access-qbjb6\") pod \"seaweedfs-64568bcd49-wfh2z\" (UID: \"a9fe57fc-d963-420a-8ceb-25ed47e43a15\") " pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.441916 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.441807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9fe57fc-d963-420a-8ceb-25ed47e43a15-data\") pod \"seaweedfs-64568bcd49-wfh2z\" (UID: \"a9fe57fc-d963-420a-8ceb-25ed47e43a15\") " pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.542900 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.542861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjb6\" (UniqueName: \"kubernetes.io/projected/a9fe57fc-d963-420a-8ceb-25ed47e43a15-kube-api-access-qbjb6\") pod \"seaweedfs-64568bcd49-wfh2z\" (UID: \"a9fe57fc-d963-420a-8ceb-25ed47e43a15\") " pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.543090 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.542912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9fe57fc-d963-420a-8ceb-25ed47e43a15-data\") pod \"seaweedfs-64568bcd49-wfh2z\" (UID: \"a9fe57fc-d963-420a-8ceb-25ed47e43a15\") " pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.543303 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.543283    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a9fe57fc-d963-420a-8ceb-25ed47e43a15-data\") pod \"seaweedfs-64568bcd49-wfh2z\" (UID: \"a9fe57fc-d963-420a-8ceb-25ed47e43a15\") " pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.558082 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.558041    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjb6\" (UniqueName: \"kubernetes.io/projected/a9fe57fc-d963-420a-8ceb-25ed47e43a15-kube-api-access-qbjb6\") pod \"seaweedfs-64568bcd49-wfh2z\" (UID: \"a9fe57fc-d963-420a-8ceb-25ed47e43a15\") " pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.700581 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.700549    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:49.828739 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:49.828714    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wfh2z"]
Jul 08 16:39:49.830742 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:39:49.830714    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fe57fc_d963_420a_8ceb_25ed47e43a15.slice/crio-ed58877068fa3cf0437dba76f1e9a3938480b1cdc69eb9ea7bcf68be3a40f6ac WatchSource:0}: Error finding container ed58877068fa3cf0437dba76f1e9a3938480b1cdc69eb9ea7bcf68be3a40f6ac: Status 404 returned error can't find the container with id ed58877068fa3cf0437dba76f1e9a3938480b1cdc69eb9ea7bcf68be3a40f6ac
Jul 08 16:39:50.283034 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:50.283001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wfh2z" event={"ID":"a9fe57fc-d963-420a-8ceb-25ed47e43a15","Type":"ContainerStarted","Data":"ed58877068fa3cf0437dba76f1e9a3938480b1cdc69eb9ea7bcf68be3a40f6ac"}
Jul 08 16:39:54.298981 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:54.298945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wfh2z" event={"ID":"a9fe57fc-d963-420a-8ceb-25ed47e43a15","Type":"ContainerStarted","Data":"766ffa6723d08f98cb07fe6cf66795a349f948bcbccafdbe7b866d66baffe027"}
Jul 08 16:39:54.299459 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:54.299006    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:39:54.316819 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:39:54.316771    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-wfh2z" podStartSLOduration=1.555562084 podStartE2EDuration="5.316730985s" totalImagesPullingTime="3.761168901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:54.315059079 +0000 UTC m=+386.253660366" watchObservedRunningTime="2026-07-08 16:39:54.316730985 +0000 UTC m=+386.255332252"
Jul 08 16:40:00.305441 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:40:00.305409    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-wfh2z"
Jul 08 16:41:00.589642 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.589564    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-q86mf"]
Jul 08 16:41:00.591853 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.591836    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.594515 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.594490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 08 16:41:00.594973 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.594951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-4t64q\""
Jul 08 16:41:00.603490 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.603469    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-q86mf"]
Jul 08 16:41:00.697904 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.697867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77csm\" (UniqueName: \"kubernetes.io/projected/b15bb9e6-16e4-473f-b909-f5902f0d2aaf-kube-api-access-77csm\") pod \"odh-model-controller-84b54b7f97-q86mf\" (UID: \"b15bb9e6-16e4-473f-b909-f5902f0d2aaf\") " pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.698101 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.697967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b15bb9e6-16e4-473f-b909-f5902f0d2aaf-cert\") pod \"odh-model-controller-84b54b7f97-q86mf\" (UID: \"b15bb9e6-16e4-473f-b909-f5902f0d2aaf\") " pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.798823 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.798764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77csm\" (UniqueName: \"kubernetes.io/projected/b15bb9e6-16e4-473f-b909-f5902f0d2aaf-kube-api-access-77csm\") pod \"odh-model-controller-84b54b7f97-q86mf\" (UID: \"b15bb9e6-16e4-473f-b909-f5902f0d2aaf\") " pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.799021 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.798897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b15bb9e6-16e4-473f-b909-f5902f0d2aaf-cert\") pod \"odh-model-controller-84b54b7f97-q86mf\" (UID: \"b15bb9e6-16e4-473f-b909-f5902f0d2aaf\") " pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.803011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.802985    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b15bb9e6-16e4-473f-b909-f5902f0d2aaf-cert\") pod \"odh-model-controller-84b54b7f97-q86mf\" (UID: \"b15bb9e6-16e4-473f-b909-f5902f0d2aaf\") " pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.811962 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.811938    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77csm\" (UniqueName: \"kubernetes.io/projected/b15bb9e6-16e4-473f-b909-f5902f0d2aaf-kube-api-access-77csm\") pod \"odh-model-controller-84b54b7f97-q86mf\" (UID: \"b15bb9e6-16e4-473f-b909-f5902f0d2aaf\") " pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:00.901768 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:00.901740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:01.026925 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:01.026899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-q86mf"]
Jul 08 16:41:01.029037 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:41:01.029010    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15bb9e6_16e4_473f_b909_f5902f0d2aaf.slice/crio-077f6b2757e0709408581dde7dc0ca0e71721bd15941745331249904ecc9c31a WatchSource:0}: Error finding container 077f6b2757e0709408581dde7dc0ca0e71721bd15941745331249904ecc9c31a: Status 404 returned error can't find the container with id 077f6b2757e0709408581dde7dc0ca0e71721bd15941745331249904ecc9c31a
Jul 08 16:41:01.540342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:01.540307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-q86mf" event={"ID":"b15bb9e6-16e4-473f-b909-f5902f0d2aaf","Type":"ContainerStarted","Data":"077f6b2757e0709408581dde7dc0ca0e71721bd15941745331249904ecc9c31a"}
Jul 08 16:41:04.555431 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:04.555395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-q86mf" event={"ID":"b15bb9e6-16e4-473f-b909-f5902f0d2aaf","Type":"ContainerStarted","Data":"4c2cd8d1c6489ff125515b894ee47d40e8316fafa2507caffbfb781015dfb1f2"}
Jul 08 16:41:04.555952 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:04.555507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:04.578170 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:04.578099    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-q86mf" podStartSLOduration=1.602707002 podStartE2EDuration="4.578055558s" totalImagesPullingTime="2.975348556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:04.575229074 +0000 UTC m=+456.513830355" watchObservedRunningTime="2026-07-08 16:41:04.578055558 +0000 UTC m=+456.516656829"
Jul 08 16:41:15.561570 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:15.561541    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-q86mf"
Jul 08 16:41:16.412413 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.412380    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4ndmt"]
Jul 08 16:41:16.414482 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.414467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.423532 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.423508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4ndmt"]
Jul 08 16:41:16.510578 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.510540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328062d5-6793-4964-971f-d4c325e3fd59-model-cache\") pod \"s3-init-4ndmt\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") " pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.510801 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.510592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x96k7\" (UniqueName: \"kubernetes.io/projected/328062d5-6793-4964-971f-d4c325e3fd59-kube-api-access-x96k7\") pod \"s3-init-4ndmt\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") " pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.611111 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.611081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328062d5-6793-4964-971f-d4c325e3fd59-model-cache\") pod \"s3-init-4ndmt\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") " pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.611481 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.611119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x96k7\" (UniqueName: \"kubernetes.io/projected/328062d5-6793-4964-971f-d4c325e3fd59-kube-api-access-x96k7\") pod \"s3-init-4ndmt\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") " pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.611518 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.611485    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328062d5-6793-4964-971f-d4c325e3fd59-model-cache\") pod \"s3-init-4ndmt\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") " pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.621833 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.621805    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x96k7\" (UniqueName: \"kubernetes.io/projected/328062d5-6793-4964-971f-d4c325e3fd59-kube-api-access-x96k7\") pod \"s3-init-4ndmt\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") " pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.723833 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.723746    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4ndmt"
Jul 08 16:41:16.856430 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:16.856302    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4ndmt"]
Jul 08 16:41:16.858881 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:41:16.858854    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod328062d5_6793_4964_971f_d4c325e3fd59.slice/crio-7c6174d103e0c0ff298a7cb410183797040464395fd0b3463bfd72cd7e18eaa3 WatchSource:0}: Error finding container 7c6174d103e0c0ff298a7cb410183797040464395fd0b3463bfd72cd7e18eaa3: Status 404 returned error can't find the container with id 7c6174d103e0c0ff298a7cb410183797040464395fd0b3463bfd72cd7e18eaa3
Jul 08 16:41:17.602953 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:17.602915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4ndmt" event={"ID":"328062d5-6793-4964-971f-d4c325e3fd59","Type":"ContainerStarted","Data":"7c6174d103e0c0ff298a7cb410183797040464395fd0b3463bfd72cd7e18eaa3"}
Jul 08 16:41:22.625974 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:22.625931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4ndmt" event={"ID":"328062d5-6793-4964-971f-d4c325e3fd59","Type":"ContainerStarted","Data":"97d8f4d55552ade8924edcf4abac05e7bb694e1f351910460cb51ddcb8d3fc2c"}
Jul 08 16:41:24.634575 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:24.634542    2561 generic.go:356] "Generic (PLEG): container finished" podID="328062d5-6793-4964-971f-d4c325e3fd59" containerID="97d8f4d55552ade8924edcf4abac05e7bb694e1f351910460cb51ddcb8d3fc2c" exitCode=0
Jul 08 16:41:24.634984 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:24.634614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4ndmt" event={"ID":"328062d5-6793-4964-971f-d4c325e3fd59","Type":"ContainerDied","Data":"97d8f4d55552ade8924edcf4abac05e7bb694e1f351910460cb51ddcb8d3fc2c"}
Jul 08 16:41:29.656341 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:29.656302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4ndmt" event={"ID":"328062d5-6793-4964-971f-d4c325e3fd59","Type":"ContainerStarted","Data":"389c2fa397471f8028516d5df43679fb058f25e90fda4e6686a68176c324dfed"}
Jul 08 16:41:29.677696 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:41:29.677626    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-4ndmt" podStartSLOduration=2.068475221 podStartE2EDuration="13.677609484s" totalImagesPullingTime="9.706275352s" totalInitContainerRuntime="1.902858911s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:29.676485221 +0000 UTC m=+481.615086506" watchObservedRunningTime="2026-07-08 16:41:29.677609484 +0000 UTC m=+481.616210755"
Jul 08 16:42:34.898056 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:34.898020    2561 generic.go:356] "Generic (PLEG): container finished" podID="328062d5-6793-4964-971f-d4c325e3fd59" containerID="389c2fa397471f8028516d5df43679fb058f25e90fda4e6686a68176c324dfed" exitCode=0
Jul 08 16:42:34.898501 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:34.898095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4ndmt" event={"ID":"328062d5-6793-4964-971f-d4c325e3fd59","Type":"ContainerDied","Data":"389c2fa397471f8028516d5df43679fb058f25e90fda4e6686a68176c324dfed"}
Jul 08 16:42:36.025487 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.025461    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4ndmt"
Jul 08 16:42:36.066738 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.066710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x96k7\" (UniqueName: \"kubernetes.io/projected/328062d5-6793-4964-971f-d4c325e3fd59-kube-api-access-x96k7\") pod \"328062d5-6793-4964-971f-d4c325e3fd59\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") "
Jul 08 16:42:36.066898 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.066752    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328062d5-6793-4964-971f-d4c325e3fd59-model-cache\") pod \"328062d5-6793-4964-971f-d4c325e3fd59\" (UID: \"328062d5-6793-4964-971f-d4c325e3fd59\") "
Jul 08 16:42:36.068802 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.068775    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328062d5-6793-4964-971f-d4c325e3fd59-kube-api-access-x96k7" (OuterVolumeSpecName: "kube-api-access-x96k7") pod "328062d5-6793-4964-971f-d4c325e3fd59" (UID: "328062d5-6793-4964-971f-d4c325e3fd59"). InnerVolumeSpecName "kube-api-access-x96k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:36.085358 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.085323    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328062d5-6793-4964-971f-d4c325e3fd59-model-cache" (OuterVolumeSpecName: "model-cache") pod "328062d5-6793-4964-971f-d4c325e3fd59" (UID: "328062d5-6793-4964-971f-d4c325e3fd59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:36.167946 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.167862    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x96k7\" (UniqueName: \"kubernetes.io/projected/328062d5-6793-4964-971f-d4c325e3fd59-kube-api-access-x96k7\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:42:36.167946 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.167893    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328062d5-6793-4964-971f-d4c325e3fd59-model-cache\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:42:36.907608 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.907581    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4ndmt"
Jul 08 16:42:36.907608 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.907586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4ndmt" event={"ID":"328062d5-6793-4964-971f-d4c325e3fd59","Type":"ContainerDied","Data":"7c6174d103e0c0ff298a7cb410183797040464395fd0b3463bfd72cd7e18eaa3"}
Jul 08 16:42:36.907822 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:36.907630    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6174d103e0c0ff298a7cb410183797040464395fd0b3463bfd72cd7e18eaa3"
Jul 08 16:42:49.119347 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.119315    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"]
Jul 08 16:42:49.119736 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.119610    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328062d5-6793-4964-971f-d4c325e3fd59" containerName="s3-init"
Jul 08 16:42:49.119736 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.119622    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="328062d5-6793-4964-971f-d4c325e3fd59" containerName="s3-init"
Jul 08 16:42:49.119736 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.119633    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328062d5-6793-4964-971f-d4c325e3fd59" containerName="download-hf-model"
Jul 08 16:42:49.119736 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.119639    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="328062d5-6793-4964-971f-d4c325e3fd59" containerName="download-hf-model"
Jul 08 16:42:49.119736 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.119711    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="328062d5-6793-4964-971f-d4c325e3fd59" containerName="s3-init"
Jul 08 16:42:49.122019 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.122002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.124931 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.124906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:42:49.125066 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.125033    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:42:49.125066 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.125033    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 08 16:42:49.125182 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.125035    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 08 16:42:49.125232 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.125183    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:42:49.125382 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.125370    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-r4bjx\""
Jul 08 16:42:49.126298 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.126285    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:42:49.134561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.134537    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"]
Jul 08 16:42:49.279161 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.279122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.279161 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.279164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.279375 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.279200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwzvn\" (UniqueName: \"kubernetes.io/projected/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-api-access-lwzvn\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.279375 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.279292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.279375 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.279329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.380584 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.380500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.380584 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.380543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.380808 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.380665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.380808 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.380712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.380915 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:42:49.380806    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 08 16:42:49.380915 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.380809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwzvn\" (UniqueName: \"kubernetes.io/projected/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-api-access-lwzvn\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.380915 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:42:49.380881    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls podName:6d0019fc-a462-41c4-a14f-05ce0b81ef57 nodeName:}" failed. No retries permitted until 2026-07-08 16:42:49.880856278 +0000 UTC m=+561.819457541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls") pod "isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" (UID: "6d0019fc-a462-41c4-a14f-05ce0b81ef57") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 08 16:42:49.381121 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.381046    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.381299 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.381279    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.381403 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.381383    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.390409 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.390389    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwzvn\" (UniqueName: \"kubernetes.io/projected/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-api-access-lwzvn\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.395029 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.395008    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"]
Jul 08 16:42:49.397756 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.397740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.400334 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.400320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 08 16:42:49.400951 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.400938    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 08 16:42:49.411333 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.411312    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"]
Jul 08 16:42:49.481661 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.481611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.481661 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.481663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e60c051-a083-4192-b735-3f490171c256-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.481915 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.481768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765mt\" (UniqueName: \"kubernetes.io/projected/8e60c051-a083-4192-b735-3f490171c256-kube-api-access-765mt\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.481915 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.481857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.481915 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.481885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.483218 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.483198    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"]
Jul 08 16:42:49.485735 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.485718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.488541 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.488515    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ed4fa-predictor-serving-cert\""
Jul 08 16:42:49.488541 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.488532    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ed4fa-kube-rbac-proxy-sar-config\""
Jul 08 16:42:49.496990 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.496969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"]
Jul 08 16:42:49.583317 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.583317 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e60c051-a083-4192-b735-3f490171c256-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.583542 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.583542 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765mt\" (UniqueName: \"kubernetes.io/projected/8e60c051-a083-4192-b735-3f490171c256-kube-api-access-765mt\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.583542 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdk9z\" (UniqueName: \"kubernetes.io/projected/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-api-access-kdk9z\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.583542 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.583759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.583759 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:42:49.583609    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 08 16:42:49.583759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.583759 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:42:49.583711    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls podName:8e60c051-a083-4192-b735-3f490171c256 nodeName:}" failed. No retries permitted until 2026-07-08 16:42:50.083663107 +0000 UTC m=+562.022264359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-6kcb8" (UID: "8e60c051-a083-4192-b735-3f490171c256") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
Jul 08 16:42:49.583924 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.583905    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e60c051-a083-4192-b735-3f490171c256-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.584278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.584260    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.584312 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.584263    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.592850 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.592828    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765mt\" (UniqueName: \"kubernetes.io/projected/8e60c051-a083-4192-b735-3f490171c256-kube-api-access-765mt\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:49.623478 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.623450    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"]
Jul 08 16:42:49.626267 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.626251    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.629063 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.629045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 08 16:42:49.629181 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.629118    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 08 16:42:49.639117 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.639096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"]
Jul 08 16:42:49.684916 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.684884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.685081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.684958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.685081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.684986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdk9z\" (UniqueName: \"kubernetes.io/projected/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-api-access-kdk9z\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.685081 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:42:49.685033    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-serving-cert: secret "error-404-isvc-ed4fa-predictor-serving-cert" not found
Jul 08 16:42:49.685236 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:42:49.685108    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls podName:845cf57f-2f64-42d4-a113-a29138a8f27d nodeName:}" failed. No retries permitted until 2026-07-08 16:42:50.185087001 +0000 UTC m=+562.123688260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls") pod "error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" (UID: "845cf57f-2f64-42d4-a113-a29138a8f27d") : secret "error-404-isvc-ed4fa-predictor-serving-cert" not found
Jul 08 16:42:49.685734 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.685712    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.694111 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.694091    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdk9z\" (UniqueName: \"kubernetes.io/projected/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-api-access-kdk9z\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:49.785481 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.785446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a165d9e-b6f5-447d-be23-b27715bea795-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.785632 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.785520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.785632 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.785585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp4r\" (UniqueName: \"kubernetes.io/projected/7a165d9e-b6f5-447d-be23-b27715bea795-kube-api-access-pbp4r\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.785632 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.785621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a165d9e-b6f5-447d-be23-b27715bea795-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.785787 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.785674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.886368 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp4r\" (UniqueName: \"kubernetes.io/projected/7a165d9e-b6f5-447d-be23-b27715bea795-kube-api-access-pbp4r\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.886368 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a165d9e-b6f5-447d-be23-b27715bea795-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.886575 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.886575 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.886575 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a165d9e-b6f5-447d-be23-b27715bea795-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.886575 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.887011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.886984    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a165d9e-b6f5-447d-be23-b27715bea795-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.887244 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.887222    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.887388 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.887368    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.889015 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.888994    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a165d9e-b6f5-447d-be23-b27715bea795-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.889015 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.889008    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-85d568d89d-j22lt\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:49.895324 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.895303    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp4r\" (UniqueName: \"kubernetes.io/projected/7a165d9e-b6f5-447d-be23-b27715bea795-kube-api-access-pbp4r\") pod \"isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:49.936230 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:49.936195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:42:50.032169 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.032132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:42:50.069438 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.069411    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"]
Jul 08 16:42:50.072800 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:42:50.072771    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a165d9e_b6f5_447d_be23_b27715bea795.slice/crio-1e1b6699718ad07cb076c325f0917a46aec9f3c084ab1d1cab7abe38934a4c0c WatchSource:0}: Error finding container 1e1b6699718ad07cb076c325f0917a46aec9f3c084ab1d1cab7abe38934a4c0c: Status 404 returned error can't find the container with id 1e1b6699718ad07cb076c325f0917a46aec9f3c084ab1d1cab7abe38934a4c0c
Jul 08 16:42:50.092374 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.092348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:50.095428 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.095401    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6kcb8\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:50.174917 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.174892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"]
Jul 08 16:42:50.177798 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:42:50.177768    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0019fc_a462_41c4_a14f_05ce0b81ef57.slice/crio-90e9d1e67938f6d5e216bef3446c320fa15d0a2422053db9063b37ae87c275a1 WatchSource:0}: Error finding container 90e9d1e67938f6d5e216bef3446c320fa15d0a2422053db9063b37ae87c275a1: Status 404 returned error can't find the container with id 90e9d1e67938f6d5e216bef3446c320fa15d0a2422053db9063b37ae87c275a1
Jul 08 16:42:50.193820 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.193797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:50.196072 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.196050    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls\") pod \"error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") " pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:50.307721 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.307675    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:42:50.398576 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.398234    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:42:50.460930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.460748    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"]
Jul 08 16:42:50.464775 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:42:50.464732    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e60c051_a083_4192_b735_3f490171c256.slice/crio-efd5cadb6e8f95e69455a2767a74168a9fe13e148807dd7c1baf8c0fbd234ff6 WatchSource:0}: Error finding container efd5cadb6e8f95e69455a2767a74168a9fe13e148807dd7c1baf8c0fbd234ff6: Status 404 returned error can't find the container with id efd5cadb6e8f95e69455a2767a74168a9fe13e148807dd7c1baf8c0fbd234ff6
Jul 08 16:42:50.562243 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.562206    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"]
Jul 08 16:42:50.566000 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:42:50.565970    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845cf57f_2f64_42d4_a113_a29138a8f27d.slice/crio-a8f3ec02d5e4c1db93281e79ddc15f31f110be8d141e8f61cabff0c63a15d586 WatchSource:0}: Error finding container a8f3ec02d5e4c1db93281e79ddc15f31f110be8d141e8f61cabff0c63a15d586: Status 404 returned error can't find the container with id a8f3ec02d5e4c1db93281e79ddc15f31f110be8d141e8f61cabff0c63a15d586
Jul 08 16:42:50.961262 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.961223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerStarted","Data":"a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f"}
Jul 08 16:42:50.961538 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.961299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerStarted","Data":"1e1b6699718ad07cb076c325f0917a46aec9f3c084ab1d1cab7abe38934a4c0c"}
Jul 08 16:42:50.962891 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.962865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerStarted","Data":"0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86"}
Jul 08 16:42:50.962981 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.962928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerStarted","Data":"efd5cadb6e8f95e69455a2767a74168a9fe13e148807dd7c1baf8c0fbd234ff6"}
Jul 08 16:42:50.964063 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.964042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" event={"ID":"845cf57f-2f64-42d4-a113-a29138a8f27d","Type":"ContainerStarted","Data":"a8f3ec02d5e4c1db93281e79ddc15f31f110be8d141e8f61cabff0c63a15d586"}
Jul 08 16:42:50.965603 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.965581    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerStarted","Data":"2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd"}
Jul 08 16:42:50.965703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:50.965633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerStarted","Data":"90e9d1e67938f6d5e216bef3446c320fa15d0a2422053db9063b37ae87c275a1"}
Jul 08 16:42:55.998358 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:55.998322    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerID="2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd" exitCode=0
Jul 08 16:42:55.998780 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:55.998405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerDied","Data":"2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd"}
Jul 08 16:42:56.001326 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:56.001300    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a165d9e-b6f5-447d-be23-b27715bea795" containerID="a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f" exitCode=0
Jul 08 16:42:56.001441 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:56.001394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerDied","Data":"a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f"}
Jul 08 16:42:56.004026 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:56.003653    2561 generic.go:356] "Generic (PLEG): container finished" podID="8e60c051-a083-4192-b735-3f490171c256" containerID="0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86" exitCode=0
Jul 08 16:42:56.004026 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:42:56.003707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerDied","Data":"0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86"}
Jul 08 16:43:23.155900 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:23.155859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerStarted","Data":"94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf"}
Jul 08 16:43:23.157732 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:23.157706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" event={"ID":"845cf57f-2f64-42d4-a113-a29138a8f27d","Type":"ContainerStarted","Data":"4d5a3fd997b56be63b50f93502d4deeca3dfac08738076916f60088c1358fd64"}
Jul 08 16:43:23.159759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:23.159738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerStarted","Data":"de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b"}
Jul 08 16:43:24.174001 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:24.173956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerStarted","Data":"60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031"}
Jul 08 16:43:26.185445 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.185396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerStarted","Data":"3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda"}
Jul 08 16:43:26.185931 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.185601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:43:26.185931 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.185770    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:43:26.187325 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.187284    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:43:26.187647 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.187614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerStarted","Data":"c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0"}
Jul 08 16:43:26.187770 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.187671    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:43:26.187770 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.187723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:43:26.188812 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.188784    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:43:26.189405 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.189386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" event={"ID":"845cf57f-2f64-42d4-a113-a29138a8f27d","Type":"ContainerStarted","Data":"e6f7c0ed67cec3c6f21fe53afbcb5d41485d5b0f6c4524621eb884b0c4495e6b"}
Jul 08 16:43:26.189607 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.189524    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:43:26.189607 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.189565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:43:26.190439 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.190417    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:43:26.191568 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.191549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerStarted","Data":"5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774"}
Jul 08 16:43:26.191775 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.191760    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:43:26.191886 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.191787    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:43:26.192674 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.192653    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:43:26.207046 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.206996    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podStartSLOduration=12.503203804 podStartE2EDuration="37.206978804s" totalImagesPullingTime="21.103944369s" totalInitContainerRuntime="3.599830631s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:26.206040702 +0000 UTC m=+598.144641989" watchObservedRunningTime="2026-07-08 16:43:26.206978804 +0000 UTC m=+598.145580073"
Jul 08 16:43:26.233558 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.233512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podStartSLOduration=4.50641747 podStartE2EDuration="37.233499383s" totalImagesPullingTime="29.298888835s" totalInitContainerRuntime="3.428193078s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:26.23235466 +0000 UTC m=+598.170955966" watchObservedRunningTime="2026-07-08 16:43:26.233499383 +0000 UTC m=+598.172100654"
Jul 08 16:43:26.255050 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.255001    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podStartSLOduration=12.549801647 podStartE2EDuration="37.254986799s" totalImagesPullingTime="21.039782276s" totalInitContainerRuntime="3.665402876s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:26.254093844 +0000 UTC m=+598.192695137" watchObservedRunningTime="2026-07-08 16:43:26.254986799 +0000 UTC m=+598.193588068"
Jul 08 16:43:26.272585 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:26.272536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podStartSLOduration=10.70138555 podStartE2EDuration="37.272522238s" totalImagesPullingTime="26.571136688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:26.272111342 +0000 UTC m=+598.210712639" watchObservedRunningTime="2026-07-08 16:43:26.272522238 +0000 UTC m=+598.211123507"
Jul 08 16:43:27.195887 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:27.195848    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:43:27.196385 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:27.195848    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:43:27.196385 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:27.196171    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:43:27.196385 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:27.196321    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:43:32.200854 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.200821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:43:32.201348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.201064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:43:32.201348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.201209    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:43:32.201435 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.201396    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:43:32.201603 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.201578    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:43:32.201748 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.201642    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:43:32.201876 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.201854    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:43:32.202319 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:32.202300    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:43:42.202185 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:42.202138    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:43:42.204641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:42.202140    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:43:42.204641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:42.202273    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:43:42.204641 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:42.202141    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:43:52.201438 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:52.201354    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:43:52.201438 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:52.201370    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:43:52.201955 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:52.201823    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:43:52.203011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:43:52.202992    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:44:02.201759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:02.201713    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:44:02.202241 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:02.201761    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:44:02.202241 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:02.201823    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:44:02.202241 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:02.202232    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:44:12.204795 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:12.204751    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:44:12.205355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:12.204801    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:44:12.205355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:12.204806    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:44:12.205355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:12.204825    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:44:22.202211 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:22.202166    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:44:22.202818 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:22.202183    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:44:22.202818 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:22.202277    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:44:22.202924 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:22.202848    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:44:32.201890 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:32.201856    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:44:32.202403 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:32.202039    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:44:32.202403 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:32.202318    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:44:39.477158 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.477123    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"]
Jul 08 16:44:39.477674 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.477366    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" containerID="cri-o://4d5a3fd997b56be63b50f93502d4deeca3dfac08738076916f60088c1358fd64" gracePeriod=30
Jul 08 16:44:39.477674 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.477423    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kube-rbac-proxy" containerID="cri-o://e6f7c0ed67cec3c6f21fe53afbcb5d41485d5b0f6c4524621eb884b0c4495e6b" gracePeriod=30
Jul 08 16:44:39.511160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.511133    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"]
Jul 08 16:44:39.513727 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.513706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.516587 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.516562    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8ceba-predictor-serving-cert\""
Jul 08 16:44:39.516861 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.516845    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8ceba-kube-rbac-proxy-sar-config\""
Jul 08 16:44:39.523749 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.523731    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"]
Jul 08 16:44:39.654857 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.654818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3c40fc-3656-4905-a966-b6dac3528ffb-proxy-tls\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.655038 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.654868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnpg\" (UniqueName: \"kubernetes.io/projected/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-api-access-xsnpg\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.655038 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.654897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.755717 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.755611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3c40fc-3656-4905-a966-b6dac3528ffb-proxy-tls\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.755717 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.755663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnpg\" (UniqueName: \"kubernetes.io/projected/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-api-access-xsnpg\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.755717 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.755712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.756599 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.756567    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.758351 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.758324    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3c40fc-3656-4905-a966-b6dac3528ffb-proxy-tls\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.767090 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.767066    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnpg\" (UniqueName: \"kubernetes.io/projected/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-api-access-xsnpg\") pod \"error-404-isvc-8ceba-predictor-84f96f9559-tdxw6\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") " pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.824239 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.824205    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:39.951279 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:39.951258    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"]
Jul 08 16:44:39.953309 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:44:39.953284    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3c40fc_3656_4905_a966_b6dac3528ffb.slice/crio-e81aa8bc60b5b82c19d85ab4672726f355892e52b1fa3ede49e658fd85e7ff20 WatchSource:0}: Error finding container e81aa8bc60b5b82c19d85ab4672726f355892e52b1fa3ede49e658fd85e7ff20: Status 404 returned error can't find the container with id e81aa8bc60b5b82c19d85ab4672726f355892e52b1fa3ede49e658fd85e7ff20
Jul 08 16:44:40.506500 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.506469    2561 generic.go:356] "Generic (PLEG): container finished" podID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerID="e6f7c0ed67cec3c6f21fe53afbcb5d41485d5b0f6c4524621eb884b0c4495e6b" exitCode=2
Jul 08 16:44:40.506910 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.506546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" event={"ID":"845cf57f-2f64-42d4-a113-a29138a8f27d","Type":"ContainerDied","Data":"e6f7c0ed67cec3c6f21fe53afbcb5d41485d5b0f6c4524621eb884b0c4495e6b"}
Jul 08 16:44:40.508269 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.508251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" event={"ID":"1c3c40fc-3656-4905-a966-b6dac3528ffb","Type":"ContainerStarted","Data":"3eb519d25693d1d79d53cbe1093549eb07401f295d79c9a20c209f49bd51a572"}
Jul 08 16:44:40.508320 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.508302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" event={"ID":"1c3c40fc-3656-4905-a966-b6dac3528ffb","Type":"ContainerStarted","Data":"9ebaefb1b4b1a9c980e6e90cf5d56faa1edda369f51c00a4b50d89527d775c67"}
Jul 08 16:44:40.508358 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.508333    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:40.508395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.508364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" event={"ID":"1c3c40fc-3656-4905-a966-b6dac3528ffb","Type":"ContainerStarted","Data":"e81aa8bc60b5b82c19d85ab4672726f355892e52b1fa3ede49e658fd85e7ff20"}
Jul 08 16:44:40.508395 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.508386    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:40.509764 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.509741    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:40.530831 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:40.530786    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podStartSLOduration=1.5307711400000001 podStartE2EDuration="1.53077114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:40.530046414 +0000 UTC m=+672.468647685" watchObservedRunningTime="2026-07-08 16:44:40.53077114 +0000 UTC m=+672.469372409"
Jul 08 16:44:41.512448 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:41.512411    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:42.196898 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.196855    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 08 16:44:42.201405 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.201375    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 08 16:44:42.202548 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.202529    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:44:42.203348 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.203332    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:44:42.518419 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.518390    2561 generic.go:356] "Generic (PLEG): container finished" podID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerID="4d5a3fd997b56be63b50f93502d4deeca3dfac08738076916f60088c1358fd64" exitCode=0
Jul 08 16:44:42.518822 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.518464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" event={"ID":"845cf57f-2f64-42d4-a113-a29138a8f27d","Type":"ContainerDied","Data":"4d5a3fd997b56be63b50f93502d4deeca3dfac08738076916f60088c1358fd64"}
Jul 08 16:44:42.628245 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.628223    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:44:42.788798 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.788701    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-rbac-proxy-sar-config\") pod \"845cf57f-2f64-42d4-a113-a29138a8f27d\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") "
Jul 08 16:44:42.788798 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.788786    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls\") pod \"845cf57f-2f64-42d4-a113-a29138a8f27d\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") "
Jul 08 16:44:42.788798 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.788808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdk9z\" (UniqueName: \"kubernetes.io/projected/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-api-access-kdk9z\") pod \"845cf57f-2f64-42d4-a113-a29138a8f27d\" (UID: \"845cf57f-2f64-42d4-a113-a29138a8f27d\") "
Jul 08 16:44:42.789135 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.789108    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "845cf57f-2f64-42d4-a113-a29138a8f27d" (UID: "845cf57f-2f64-42d4-a113-a29138a8f27d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:44:42.790925 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.790905    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-api-access-kdk9z" (OuterVolumeSpecName: "kube-api-access-kdk9z") pod "845cf57f-2f64-42d4-a113-a29138a8f27d" (UID: "845cf57f-2f64-42d4-a113-a29138a8f27d"). InnerVolumeSpecName "kube-api-access-kdk9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:44:42.791041 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.791021    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "845cf57f-2f64-42d4-a113-a29138a8f27d" (UID: "845cf57f-2f64-42d4-a113-a29138a8f27d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:44:42.889922 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.889883    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/845cf57f-2f64-42d4-a113-a29138a8f27d-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:44:42.889922 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.889915    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdk9z\" (UniqueName: \"kubernetes.io/projected/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-api-access-kdk9z\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:44:42.889922 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:42.889931    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/845cf57f-2f64-42d4-a113-a29138a8f27d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:44:43.525064 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:43.525032    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"
Jul 08 16:44:43.525064 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:43.525046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9" event={"ID":"845cf57f-2f64-42d4-a113-a29138a8f27d","Type":"ContainerDied","Data":"a8f3ec02d5e4c1db93281e79ddc15f31f110be8d141e8f61cabff0c63a15d586"}
Jul 08 16:44:43.525611 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:43.525105    2561 scope.go:117] "RemoveContainer" containerID="e6f7c0ed67cec3c6f21fe53afbcb5d41485d5b0f6c4524621eb884b0c4495e6b"
Jul 08 16:44:43.533090 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:43.533073    2561 scope.go:117] "RemoveContainer" containerID="4d5a3fd997b56be63b50f93502d4deeca3dfac08738076916f60088c1358fd64"
Jul 08 16:44:43.544816 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:43.544792    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"]
Jul 08 16:44:43.548703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:43.548668    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ed4fa-predictor-695ddbc98f-tbbv9"]
Jul 08 16:44:44.801897 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:44.801868    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" path="/var/lib/kubelet/pods/845cf57f-2f64-42d4-a113-a29138a8f27d/volumes"
Jul 08 16:44:46.518338 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:46.518310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:44:46.518922 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:46.518893    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:44:56.519536 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:44:56.519497    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:45:06.519307 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:06.519266    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:45:16.519568 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:16.519525    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:45:19.306331 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.306253    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"]
Jul 08 16:45:19.306780 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.306567    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" containerID="cri-o://94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf" gracePeriod=30
Jul 08 16:45:19.306780 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.306607    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kube-rbac-proxy" containerID="cri-o://3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda" gracePeriod=30
Jul 08 16:45:19.370221 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.370187    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"]
Jul 08 16:45:19.370441 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.370412    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" containerID="cri-o://de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b" gracePeriod=30
Jul 08 16:45:19.370571 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.370468    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kube-rbac-proxy" containerID="cri-o://5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774" gracePeriod=30
Jul 08 16:45:19.445770 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.445728    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"]
Jul 08 16:45:19.446190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.446151    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" containerID="cri-o://60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031" gracePeriod=30
Jul 08 16:45:19.446606 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.446578    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kube-rbac-proxy" containerID="cri-o://c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0" gracePeriod=30
Jul 08 16:45:19.509392 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509353    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"]
Jul 08 16:45:19.509759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509746    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kube-rbac-proxy"
Jul 08 16:45:19.509759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509761    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kube-rbac-proxy"
Jul 08 16:45:19.509865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509790    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container"
Jul 08 16:45:19.509865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509796    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container"
Jul 08 16:45:19.509865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509851    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kube-rbac-proxy"
Jul 08 16:45:19.509952 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.509867    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="845cf57f-2f64-42d4-a113-a29138a8f27d" containerName="kserve-container"
Jul 08 16:45:19.511952 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.511936    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.514703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.514665    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fabf2-predictor-serving-cert\""
Jul 08 16:45:19.514819 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.514669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fabf2-kube-rbac-proxy-sar-config\""
Jul 08 16:45:19.522510 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.522484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"]
Jul 08 16:45:19.595251 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.595215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.595419 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.595257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4xj\" (UniqueName: \"kubernetes.io/projected/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-api-access-ht4xj\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.595419 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.595347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8240b020-722e-4446-9dc9-666e3e2b2bab-proxy-tls\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.682579 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.682549    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a165d9e-b6f5-447d-be23-b27715bea795" containerID="3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda" exitCode=2
Jul 08 16:45:19.682782 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.682620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerDied","Data":"3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda"}
Jul 08 16:45:19.684529 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.684511    2561 generic.go:356] "Generic (PLEG): container finished" podID="8e60c051-a083-4192-b735-3f490171c256" containerID="c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0" exitCode=2
Jul 08 16:45:19.684603 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.684586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerDied","Data":"c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0"}
Jul 08 16:45:19.686429 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.686408    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerID="5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774" exitCode=2
Jul 08 16:45:19.686554 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.686479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerDied","Data":"5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774"}
Jul 08 16:45:19.695870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.695845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8240b020-722e-4446-9dc9-666e3e2b2bab-proxy-tls\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.695932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.695888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.695932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.695911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4xj\" (UniqueName: \"kubernetes.io/projected/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-api-access-ht4xj\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.696598 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.696574    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.698331 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.698314    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8240b020-722e-4446-9dc9-666e3e2b2bab-proxy-tls\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.705155 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.705132    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4xj\" (UniqueName: \"kubernetes.io/projected/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-api-access-ht4xj\") pod \"error-404-isvc-fabf2-predictor-54645d4667-9pmq6\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") " pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.822108 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.822028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:19.955773 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:19.955740    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"]
Jul 08 16:45:19.957854 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:45:19.957821    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8240b020_722e_4446_9dc9_666e3e2b2bab.slice/crio-23dc877bc5c282b88b11d65012f481398d8253e3f745c5f959e72f53e5005428 WatchSource:0}: Error finding container 23dc877bc5c282b88b11d65012f481398d8253e3f745c5f959e72f53e5005428: Status 404 returned error can't find the container with id 23dc877bc5c282b88b11d65012f481398d8253e3f745c5f959e72f53e5005428
Jul 08 16:45:20.692435 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:20.692390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" event={"ID":"8240b020-722e-4446-9dc9-666e3e2b2bab","Type":"ContainerStarted","Data":"b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58"}
Jul 08 16:45:20.692858 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:20.692468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" event={"ID":"8240b020-722e-4446-9dc9-666e3e2b2bab","Type":"ContainerStarted","Data":"9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee"}
Jul 08 16:45:20.692858 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:20.692501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" event={"ID":"8240b020-722e-4446-9dc9-666e3e2b2bab","Type":"ContainerStarted","Data":"23dc877bc5c282b88b11d65012f481398d8253e3f745c5f959e72f53e5005428"}
Jul 08 16:45:20.692858 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:20.692535    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:20.714808 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:20.714755    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podStartSLOduration=1.71472787 podStartE2EDuration="1.71472787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:45:20.71358311 +0000 UTC m=+712.652184379" watchObservedRunningTime="2026-07-08 16:45:20.71472787 +0000 UTC m=+712.653329172"
Jul 08 16:45:21.696527 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:21.696497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:21.697738 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:21.697706    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:22.196048 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.196003    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 08 16:45:22.196230 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.196004    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 08 16:45:22.197263 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.197243    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Jul 08 16:45:22.201622 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.201604    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 08 16:45:22.201872 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.201852    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 08 16:45:22.203055 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.203037    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 08 16:45:22.700894 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:22.700856    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:23.399011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.398988    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:45:23.529843 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.529743    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-kube-rbac-proxy-sar-config\") pod \"8e60c051-a083-4192-b735-3f490171c256\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") "
Jul 08 16:45:23.529843 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.529796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-cabundle-cert\") pod \"8e60c051-a083-4192-b735-3f490171c256\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") "
Jul 08 16:45:23.529843 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.529839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-765mt\" (UniqueName: \"kubernetes.io/projected/8e60c051-a083-4192-b735-3f490171c256-kube-api-access-765mt\") pod \"8e60c051-a083-4192-b735-3f490171c256\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") "
Jul 08 16:45:23.530116 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.529899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls\") pod \"8e60c051-a083-4192-b735-3f490171c256\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") "
Jul 08 16:45:23.530116 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.529998    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e60c051-a083-4192-b735-3f490171c256-kserve-provision-location\") pod \"8e60c051-a083-4192-b735-3f490171c256\" (UID: \"8e60c051-a083-4192-b735-3f490171c256\") "
Jul 08 16:45:23.530228 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.530135    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e60c051-a083-4192-b735-3f490171c256" (UID: "8e60c051-a083-4192-b735-3f490171c256"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:23.530228 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.530198    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e60c051-a083-4192-b735-3f490171c256" (UID: "8e60c051-a083-4192-b735-3f490171c256"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:23.530343 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.530316    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:23.530400 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.530342    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e60c051-a083-4192-b735-3f490171c256-cabundle-cert\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:23.530400 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.530351    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e60c051-a083-4192-b735-3f490171c256-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e60c051-a083-4192-b735-3f490171c256" (UID: "8e60c051-a083-4192-b735-3f490171c256"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:45:23.531982 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.531960    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e60c051-a083-4192-b735-3f490171c256" (UID: "8e60c051-a083-4192-b735-3f490171c256"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:45:23.532049 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.532027    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e60c051-a083-4192-b735-3f490171c256-kube-api-access-765mt" (OuterVolumeSpecName: "kube-api-access-765mt") pod "8e60c051-a083-4192-b735-3f490171c256" (UID: "8e60c051-a083-4192-b735-3f490171c256"). InnerVolumeSpecName "kube-api-access-765mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:45:23.631064 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.631025    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e60c051-a083-4192-b735-3f490171c256-kserve-provision-location\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:23.631064 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.631056    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765mt\" (UniqueName: \"kubernetes.io/projected/8e60c051-a083-4192-b735-3f490171c256-kube-api-access-765mt\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:23.631064 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.631066    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e60c051-a083-4192-b735-3f490171c256-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:23.706963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.706908    2561 generic.go:356] "Generic (PLEG): container finished" podID="8e60c051-a083-4192-b735-3f490171c256" containerID="60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031" exitCode=0
Jul 08 16:45:23.706963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.706956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerDied","Data":"60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031"}
Jul 08 16:45:23.707616 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.706992    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"
Jul 08 16:45:23.707616 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.707051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8" event={"ID":"8e60c051-a083-4192-b735-3f490171c256","Type":"ContainerDied","Data":"efd5cadb6e8f95e69455a2767a74168a9fe13e148807dd7c1baf8c0fbd234ff6"}
Jul 08 16:45:23.707616 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.707080    2561 scope.go:117] "RemoveContainer" containerID="c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0"
Jul 08 16:45:23.715667 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.715647    2561 scope.go:117] "RemoveContainer" containerID="60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031"
Jul 08 16:45:23.724201 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.724177    2561 scope.go:117] "RemoveContainer" containerID="0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86"
Jul 08 16:45:23.729646 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.729625    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"]
Jul 08 16:45:23.732413 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.732390    2561 scope.go:117] "RemoveContainer" containerID="c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0"
Jul 08 16:45:23.732842 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:23.732819    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0\": container with ID starting with c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0 not found: ID does not exist" containerID="c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0"
Jul 08 16:45:23.732912 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.732861    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0"} err="failed to get container status \"c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0\": rpc error: code = NotFound desc = could not find container \"c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0\": container with ID starting with c7589f2568df5f18a8ea794f81b8a34e20913ed25996d1f567ed91b0761f63c0 not found: ID does not exist"
Jul 08 16:45:23.732912 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.732884    2561 scope.go:117] "RemoveContainer" containerID="60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031"
Jul 08 16:45:23.733165 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:23.733147    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031\": container with ID starting with 60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031 not found: ID does not exist" containerID="60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031"
Jul 08 16:45:23.733231 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.733186    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031"} err="failed to get container status \"60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031\": rpc error: code = NotFound desc = could not find container \"60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031\": container with ID starting with 60d1208aae19fc09565f58b3f68ce6fd3d62f6cade1d21e9818cc891d024e031 not found: ID does not exist"
Jul 08 16:45:23.733231 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.733209    2561 scope.go:117] "RemoveContainer" containerID="0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86"
Jul 08 16:45:23.733489 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:23.733466    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86\": container with ID starting with 0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86 not found: ID does not exist" containerID="0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86"
Jul 08 16:45:23.733543 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.733491    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86"} err="failed to get container status \"0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86\": rpc error: code = NotFound desc = could not find container \"0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86\": container with ID starting with 0704e844fca7c220a433b7bb8de87e9694669361408687d5716deb00bb674d86 not found: ID does not exist"
Jul 08 16:45:23.733793 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:23.733779    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6kcb8"]
Jul 08 16:45:24.641093 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.641068    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:45:24.713260 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.713231    2561 generic.go:356] "Generic (PLEG): container finished" podID="7a165d9e-b6f5-447d-be23-b27715bea795" containerID="94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf" exitCode=0
Jul 08 16:45:24.713748 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.713296    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"
Jul 08 16:45:24.713748 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.713305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerDied","Data":"94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf"}
Jul 08 16:45:24.713748 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.713362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r" event={"ID":"7a165d9e-b6f5-447d-be23-b27715bea795","Type":"ContainerDied","Data":"1e1b6699718ad07cb076c325f0917a46aec9f3c084ab1d1cab7abe38934a4c0c"}
Jul 08 16:45:24.713748 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.713405    2561 scope.go:117] "RemoveContainer" containerID="3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda"
Jul 08 16:45:24.721204 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.721188    2561 scope.go:117] "RemoveContainer" containerID="94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf"
Jul 08 16:45:24.728609 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.728579    2561 scope.go:117] "RemoveContainer" containerID="a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f"
Jul 08 16:45:24.735539 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.735522    2561 scope.go:117] "RemoveContainer" containerID="3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda"
Jul 08 16:45:24.735820 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:24.735792    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda\": container with ID starting with 3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda not found: ID does not exist" containerID="3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda"
Jul 08 16:45:24.735872 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.735820    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda"} err="failed to get container status \"3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda\": rpc error: code = NotFound desc = could not find container \"3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda\": container with ID starting with 3f696621c0a12a63fe767c328e6c835a639728d6e725750d602176b2ba5a4bda not found: ID does not exist"
Jul 08 16:45:24.735872 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.735837    2561 scope.go:117] "RemoveContainer" containerID="94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf"
Jul 08 16:45:24.736070 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:24.736053    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf\": container with ID starting with 94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf not found: ID does not exist" containerID="94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf"
Jul 08 16:45:24.736110 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.736076    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf"} err="failed to get container status \"94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf\": rpc error: code = NotFound desc = could not find container \"94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf\": container with ID starting with 94809e21f92ca9aa3ca160419fbdfdf83af56cdf77c292adc77baba5fd842bdf not found: ID does not exist"
Jul 08 16:45:24.736110 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.736091    2561 scope.go:117] "RemoveContainer" containerID="a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f"
Jul 08 16:45:24.736318 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:24.736303    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f\": container with ID starting with a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f not found: ID does not exist" containerID="a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f"
Jul 08 16:45:24.736367 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.736323    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f"} err="failed to get container status \"a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f\": rpc error: code = NotFound desc = could not find container \"a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f\": container with ID starting with a7cfd70bf42855436aa332e4c1a72ae7f1677ebeffed7b461e4ff36f7fb5683f not found: ID does not exist"
Jul 08 16:45:24.741576 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-kube-rbac-proxy-sar-config\") pod \"7a165d9e-b6f5-447d-be23-b27715bea795\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") "
Jul 08 16:45:24.741640 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741596    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-cabundle-cert\") pod \"7a165d9e-b6f5-447d-be23-b27715bea795\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") "
Jul 08 16:45:24.741674 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a165d9e-b6f5-447d-be23-b27715bea795-kserve-provision-location\") pod \"7a165d9e-b6f5-447d-be23-b27715bea795\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") "
Jul 08 16:45:24.741738 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741674    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a165d9e-b6f5-447d-be23-b27715bea795-proxy-tls\") pod \"7a165d9e-b6f5-447d-be23-b27715bea795\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") "
Jul 08 16:45:24.741824 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbp4r\" (UniqueName: \"kubernetes.io/projected/7a165d9e-b6f5-447d-be23-b27715bea795-kube-api-access-pbp4r\") pod \"7a165d9e-b6f5-447d-be23-b27715bea795\" (UID: \"7a165d9e-b6f5-447d-be23-b27715bea795\") "
Jul 08 16:45:24.741979 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741952    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a165d9e-b6f5-447d-be23-b27715bea795" (UID: "7a165d9e-b6f5-447d-be23-b27715bea795"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:24.742081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.741985    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a165d9e-b6f5-447d-be23-b27715bea795-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a165d9e-b6f5-447d-be23-b27715bea795" (UID: "7a165d9e-b6f5-447d-be23-b27715bea795"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:45:24.742081 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.742011    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a165d9e-b6f5-447d-be23-b27715bea795" (UID: "7a165d9e-b6f5-447d-be23-b27715bea795"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:24.742190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.742095    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a165d9e-b6f5-447d-be23-b27715bea795-kserve-provision-location\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:24.742190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.742119    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:24.742190 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.742129    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a165d9e-b6f5-447d-be23-b27715bea795-cabundle-cert\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:24.743739 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.743674    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a165d9e-b6f5-447d-be23-b27715bea795-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a165d9e-b6f5-447d-be23-b27715bea795" (UID: "7a165d9e-b6f5-447d-be23-b27715bea795"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:45:24.743813 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.743792    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a165d9e-b6f5-447d-be23-b27715bea795-kube-api-access-pbp4r" (OuterVolumeSpecName: "kube-api-access-pbp4r") pod "7a165d9e-b6f5-447d-be23-b27715bea795" (UID: "7a165d9e-b6f5-447d-be23-b27715bea795"). InnerVolumeSpecName "kube-api-access-pbp4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:45:24.801106 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.801075    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e60c051-a083-4192-b735-3f490171c256" path="/var/lib/kubelet/pods/8e60c051-a083-4192-b735-3f490171c256/volumes"
Jul 08 16:45:24.842839 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.842813    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a165d9e-b6f5-447d-be23-b27715bea795-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:24.842839 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:24.842836    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbp4r\" (UniqueName: \"kubernetes.io/projected/7a165d9e-b6f5-447d-be23-b27715bea795-kube-api-access-pbp4r\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:25.031173 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.031098    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"]
Jul 08 16:45:25.035128 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.035108    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7f5c6b4bf8-87p9r"]
Jul 08 16:45:25.196209 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.196187    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:45:25.346858 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.346768    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwzvn\" (UniqueName: \"kubernetes.io/projected/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-api-access-lwzvn\") pod \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") "
Jul 08 16:45:25.346858 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.346833    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-cabundle-cert\") pod \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") "
Jul 08 16:45:25.346858 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.346859    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kserve-provision-location\") pod \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") "
Jul 08 16:45:25.347145 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.346905    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls\") pod \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") "
Jul 08 16:45:25.347145 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.346920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-rbac-proxy-sar-config\") pod \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\" (UID: \"6d0019fc-a462-41c4-a14f-05ce0b81ef57\") "
Jul 08 16:45:25.347257 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.347192    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d0019fc-a462-41c4-a14f-05ce0b81ef57" (UID: "6d0019fc-a462-41c4-a14f-05ce0b81ef57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:45:25.347257 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.347233    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d0019fc-a462-41c4-a14f-05ce0b81ef57" (UID: "6d0019fc-a462-41c4-a14f-05ce0b81ef57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:25.347335 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.347287    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d0019fc-a462-41c4-a14f-05ce0b81ef57" (UID: "6d0019fc-a462-41c4-a14f-05ce0b81ef57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:45:25.348953 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.348930    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d0019fc-a462-41c4-a14f-05ce0b81ef57" (UID: "6d0019fc-a462-41c4-a14f-05ce0b81ef57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:45:25.349024 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.349007    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-api-access-lwzvn" (OuterVolumeSpecName: "kube-api-access-lwzvn") pod "6d0019fc-a462-41c4-a14f-05ce0b81ef57" (UID: "6d0019fc-a462-41c4-a14f-05ce0b81ef57"). InnerVolumeSpecName "kube-api-access-lwzvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:45:25.447561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.447525    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d0019fc-a462-41c4-a14f-05ce0b81ef57-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:25.447561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.447553    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:25.447561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.447563    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwzvn\" (UniqueName: \"kubernetes.io/projected/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kube-api-access-lwzvn\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:25.447811 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.447572    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d0019fc-a462-41c4-a14f-05ce0b81ef57-cabundle-cert\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:25.447811 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.447584    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d0019fc-a462-41c4-a14f-05ce0b81ef57-kserve-provision-location\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:45:25.721440 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.721405    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerID="de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b" exitCode=0
Jul 08 16:45:25.721865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.721463    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"
Jul 08 16:45:25.721865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.721467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerDied","Data":"de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b"}
Jul 08 16:45:25.721865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.721608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt" event={"ID":"6d0019fc-a462-41c4-a14f-05ce0b81ef57","Type":"ContainerDied","Data":"90e9d1e67938f6d5e216bef3446c320fa15d0a2422053db9063b37ae87c275a1"}
Jul 08 16:45:25.721865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.721650    2561 scope.go:117] "RemoveContainer" containerID="5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774"
Jul 08 16:45:25.730126 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.730110    2561 scope.go:117] "RemoveContainer" containerID="de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b"
Jul 08 16:45:25.737602 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.737582    2561 scope.go:117] "RemoveContainer" containerID="2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd"
Jul 08 16:45:25.743009 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.742987    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"]
Jul 08 16:45:25.745833 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.745812    2561 scope.go:117] "RemoveContainer" containerID="5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774"
Jul 08 16:45:25.746216 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:25.746194    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774\": container with ID starting with 5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774 not found: ID does not exist" containerID="5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774"
Jul 08 16:45:25.746291 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.746223    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774"} err="failed to get container status \"5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774\": rpc error: code = NotFound desc = could not find container \"5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774\": container with ID starting with 5a352cc9a816d2623132d4706194beb9b85ab29f39eccd419fd0f7b05f020774 not found: ID does not exist"
Jul 08 16:45:25.746291 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.746245    2561 scope.go:117] "RemoveContainer" containerID="de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b"
Jul 08 16:45:25.746520 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:25.746503    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b\": container with ID starting with de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b not found: ID does not exist" containerID="de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b"
Jul 08 16:45:25.746578 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.746526    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b"} err="failed to get container status \"de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b\": rpc error: code = NotFound desc = could not find container \"de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b\": container with ID starting with de15d1b40f686f9cf589c650cba02145cb19d6eeee0a7f55ac37bbca8f3e500b not found: ID does not exist"
Jul 08 16:45:25.746578 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.746541    2561 scope.go:117] "RemoveContainer" containerID="2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd"
Jul 08 16:45:25.746674 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.746634    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-85d568d89d-j22lt"]
Jul 08 16:45:25.746816 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:45:25.746799    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd\": container with ID starting with 2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd not found: ID does not exist" containerID="2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd"
Jul 08 16:45:25.746857 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:25.746823    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd"} err="failed to get container status \"2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd\": rpc error: code = NotFound desc = could not find container \"2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd\": container with ID starting with 2d255b95cace66c3522e9b7f5c7a0144c763c25356f0a1e1654396547c0135cd not found: ID does not exist"
Jul 08 16:45:26.519760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:26.519734    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:45:26.803053 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:26.802973    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" path="/var/lib/kubelet/pods/6d0019fc-a462-41c4-a14f-05ce0b81ef57/volumes"
Jul 08 16:45:26.803454 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:26.803441    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" path="/var/lib/kubelet/pods/7a165d9e-b6f5-447d-be23-b27715bea795/volumes"
Jul 08 16:45:27.705463 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:27.705430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:45:27.706015 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:27.705984    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:37.706118 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:37.706070    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:47.706747 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:47.706709    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:45:57.706831 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:45:57.706787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 16:46:07.706870 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:46:07.706833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:53:54.433270 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.433232    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"]
Jul 08 16:53:54.433845 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.433539    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" containerID="cri-o://9ebaefb1b4b1a9c980e6e90cf5d56faa1edda369f51c00a4b50d89527d775c67" gracePeriod=30
Jul 08 16:53:54.433845 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.433551    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kube-rbac-proxy" containerID="cri-o://3eb519d25693d1d79d53cbe1093549eb07401f295d79c9a20c209f49bd51a572" gracePeriod=30
Jul 08 16:53:54.539869 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.539834    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"]
Jul 08 16:53:54.540151 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540136    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="storage-initializer"
Jul 08 16:53:54.540151 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540150    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="storage-initializer"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540160    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540165    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540175    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540180    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540187    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540193    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540201    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540206    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540213    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="storage-initializer"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540219    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="storage-initializer"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540226    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540230    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540238    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="storage-initializer"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540243    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="storage-initializer"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540248    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container"
Jul 08 16:53:54.540272 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540253    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container"
Jul 08 16:53:54.540940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kserve-container"
Jul 08 16:53:54.540940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540315    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e60c051-a083-4192-b735-3f490171c256" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540322    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a165d9e-b6f5-447d-be23-b27715bea795" containerName="kserve-container"
Jul 08 16:53:54.540940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540328    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kube-rbac-proxy"
Jul 08 16:53:54.540940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.540332    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d0019fc-a462-41c4-a14f-05ce0b81ef57" containerName="kserve-container"
Jul 08 16:53:54.547803 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.547772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.550977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.550947    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8befa-predictor-serving-cert\""
Jul 08 16:53:54.551098 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.550959    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8befa-kube-rbac-proxy-sar-config\""
Jul 08 16:53:54.555808 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.555534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"]
Jul 08 16:53:54.631340 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.631304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.631508 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.631348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2979b4f3-c547-441d-9927-921676de4a1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.631508 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.631373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpc2\" (UniqueName: \"kubernetes.io/projected/2979b4f3-c547-441d-9927-921676de4a1d-kube-api-access-swpc2\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.704079 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.703995    2561 generic.go:356] "Generic (PLEG): container finished" podID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerID="3eb519d25693d1d79d53cbe1093549eb07401f295d79c9a20c209f49bd51a572" exitCode=2
Jul 08 16:53:54.704208 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.704073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" event={"ID":"1c3c40fc-3656-4905-a966-b6dac3528ffb","Type":"ContainerDied","Data":"3eb519d25693d1d79d53cbe1093549eb07401f295d79c9a20c209f49bd51a572"}
Jul 08 16:53:54.731889 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.731863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.731963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.731903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2979b4f3-c547-441d-9927-921676de4a1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.731963 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.731924    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpc2\" (UniqueName: \"kubernetes.io/projected/2979b4f3-c547-441d-9927-921676de4a1d-kube-api-access-swpc2\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.732076 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:53:54.732034    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8befa-predictor-serving-cert: secret "error-404-isvc-8befa-predictor-serving-cert" not found
Jul 08 16:53:54.732128 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:53:54.732116    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls podName:2979b4f3-c547-441d-9927-921676de4a1d nodeName:}" failed. No retries permitted until 2026-07-08 16:53:55.232094148 +0000 UTC m=+1227.170695415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls") pod "error-404-isvc-8befa-predictor-5957789c65-2q4tr" (UID: "2979b4f3-c547-441d-9927-921676de4a1d") : secret "error-404-isvc-8befa-predictor-serving-cert" not found
Jul 08 16:53:54.732592 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.732570    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2979b4f3-c547-441d-9927-921676de4a1d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:54.741468 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:54.741438    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpc2\" (UniqueName: \"kubernetes.io/projected/2979b4f3-c547-441d-9927-921676de4a1d-kube-api-access-swpc2\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:55.236418 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:55.236367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:55.238807 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:55.238774    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls\") pod \"error-404-isvc-8befa-predictor-5957789c65-2q4tr\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") " pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:55.460008 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:55.459970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:55.591714 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:55.591671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"]
Jul 08 16:53:55.594088 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:53:55.594052    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2979b4f3_c547_441d_9927_921676de4a1d.slice/crio-bafc6fa38503b7b4bd51c7bb846ffaf7fc8c7903ec69d417ac72ceef826b91b5 WatchSource:0}: Error finding container bafc6fa38503b7b4bd51c7bb846ffaf7fc8c7903ec69d417ac72ceef826b91b5: Status 404 returned error can't find the container with id bafc6fa38503b7b4bd51c7bb846ffaf7fc8c7903ec69d417ac72ceef826b91b5
Jul 08 16:53:55.709972 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:55.709945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" event={"ID":"2979b4f3-c547-441d-9927-921676de4a1d","Type":"ContainerStarted","Data":"6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d"}
Jul 08 16:53:55.710086 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:55.710013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" event={"ID":"2979b4f3-c547-441d-9927-921676de4a1d","Type":"ContainerStarted","Data":"bafc6fa38503b7b4bd51c7bb846ffaf7fc8c7903ec69d417ac72ceef826b91b5"}
Jul 08 16:53:56.512805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:56.512766    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 08 16:53:56.519282 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:56.519252    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 08 16:53:56.715506 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:56.715467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" event={"ID":"2979b4f3-c547-441d-9927-921676de4a1d","Type":"ContainerStarted","Data":"24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408"}
Jul 08 16:53:56.715711 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:56.715642    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:56.739333 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:56.739288    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podStartSLOduration=2.7392730309999997 podStartE2EDuration="2.739273031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:53:56.73684183 +0000 UTC m=+1228.675443097" watchObservedRunningTime="2026-07-08 16:53:56.739273031 +0000 UTC m=+1228.677874358"
Jul 08 16:53:57.723427 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.723393    2561 generic.go:356] "Generic (PLEG): container finished" podID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerID="9ebaefb1b4b1a9c980e6e90cf5d56faa1edda369f51c00a4b50d89527d775c67" exitCode=0
Jul 08 16:53:57.723834 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.723466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" event={"ID":"1c3c40fc-3656-4905-a966-b6dac3528ffb","Type":"ContainerDied","Data":"9ebaefb1b4b1a9c980e6e90cf5d56faa1edda369f51c00a4b50d89527d775c67"}
Jul 08 16:53:57.723834 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.723767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:53:57.724987 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.724958    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:53:57.776965 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.776942    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:53:57.858569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.858489    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3c40fc-3656-4905-a966-b6dac3528ffb-proxy-tls\") pod \"1c3c40fc-3656-4905-a966-b6dac3528ffb\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") "
Jul 08 16:53:57.858569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.858538    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-rbac-proxy-sar-config\") pod \"1c3c40fc-3656-4905-a966-b6dac3528ffb\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") "
Jul 08 16:53:57.858569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.858562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsnpg\" (UniqueName: \"kubernetes.io/projected/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-api-access-xsnpg\") pod \"1c3c40fc-3656-4905-a966-b6dac3528ffb\" (UID: \"1c3c40fc-3656-4905-a966-b6dac3528ffb\") "
Jul 08 16:53:57.859131 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.858938    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c3c40fc-3656-4905-a966-b6dac3528ffb" (UID: "1c3c40fc-3656-4905-a966-b6dac3528ffb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:53:57.859131 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.859075    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:53:57.860723 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.860675    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-api-access-xsnpg" (OuterVolumeSpecName: "kube-api-access-xsnpg") pod "1c3c40fc-3656-4905-a966-b6dac3528ffb" (UID: "1c3c40fc-3656-4905-a966-b6dac3528ffb"). InnerVolumeSpecName "kube-api-access-xsnpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:53:57.860940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.860919    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3c40fc-3656-4905-a966-b6dac3528ffb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c3c40fc-3656-4905-a966-b6dac3528ffb" (UID: "1c3c40fc-3656-4905-a966-b6dac3528ffb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:53:57.959719 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.959670    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3c40fc-3656-4905-a966-b6dac3528ffb-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:53:57.959719 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:57.959715    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsnpg\" (UniqueName: \"kubernetes.io/projected/1c3c40fc-3656-4905-a966-b6dac3528ffb-kube-api-access-xsnpg\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:53:58.730059 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.730028    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"
Jul 08 16:53:58.730059 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.730036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6" event={"ID":"1c3c40fc-3656-4905-a966-b6dac3528ffb","Type":"ContainerDied","Data":"e81aa8bc60b5b82c19d85ab4672726f355892e52b1fa3ede49e658fd85e7ff20"}
Jul 08 16:53:58.730789 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.730118    2561 scope.go:117] "RemoveContainer" containerID="3eb519d25693d1d79d53cbe1093549eb07401f295d79c9a20c209f49bd51a572"
Jul 08 16:53:58.730884 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.730859    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:53:58.738532 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.738511    2561 scope.go:117] "RemoveContainer" containerID="9ebaefb1b4b1a9c980e6e90cf5d56faa1edda369f51c00a4b50d89527d775c67"
Jul 08 16:53:58.757332 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.757302    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"]
Jul 08 16:53:58.762182 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.762160    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8ceba-predictor-84f96f9559-tdxw6"]
Jul 08 16:53:58.804018 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:53:58.803911    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" path="/var/lib/kubelet/pods/1c3c40fc-3656-4905-a966-b6dac3528ffb/volumes"
Jul 08 16:54:03.735544 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:03.735511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:54:03.736046 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:03.736021    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:54:13.736484 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:13.736440    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:54:23.736611 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:23.736564    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:54:33.736490 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:33.736444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 16:54:34.264041 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.264006    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"]
Jul 08 16:54:34.264324 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.264295    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container" containerID="cri-o://9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee" gracePeriod=30
Jul 08 16:54:34.264423 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.264393    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kube-rbac-proxy" containerID="cri-o://b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58" gracePeriod=30
Jul 08 16:54:34.297015 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.296989    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"]
Jul 08 16:54:34.297302 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.297290    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container"
Jul 08 16:54:34.297342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.297304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container"
Jul 08 16:54:34.297342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.297329    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kube-rbac-proxy"
Jul 08 16:54:34.297342 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.297335    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kube-rbac-proxy"
Jul 08 16:54:34.297433 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.297381    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kserve-container"
Jul 08 16:54:34.297433 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.297393    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3c40fc-3656-4905-a966-b6dac3528ffb" containerName="kube-rbac-proxy"
Jul 08 16:54:34.301257 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.301241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.303861 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.303839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3816e-kube-rbac-proxy-sar-config\""
Jul 08 16:54:34.303966 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.303850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3816e-predictor-serving-cert\""
Jul 08 16:54:34.311308 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.311284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"]
Jul 08 16:54:34.367779 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.367740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.367779 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.367776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbd4\" (UniqueName: \"kubernetes.io/projected/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-api-access-lnbd4\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.368003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.367800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.468347 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.468308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbd4\" (UniqueName: \"kubernetes.io/projected/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-api-access-lnbd4\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.468347 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.468347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.468569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.468431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.468569 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:54:34.468520    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3816e-predictor-serving-cert: secret "error-404-isvc-3816e-predictor-serving-cert" not found
Jul 08 16:54:34.468706 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:54:34.468581    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls podName:bfe92b54-88f7-44a6-aa95-a0d49fbf63d5 nodeName:}" failed. No retries permitted until 2026-07-08 16:54:34.968563509 +0000 UTC m=+1266.907164767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls") pod "error-404-isvc-3816e-predictor-65dc76b695-ckgw9" (UID: "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5") : secret "error-404-isvc-3816e-predictor-serving-cert" not found
Jul 08 16:54:34.469105 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.469080    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.478840 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.478820    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbd4\" (UniqueName: \"kubernetes.io/projected/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-api-access-lnbd4\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.873560 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.873528    2561 generic.go:356] "Generic (PLEG): container finished" podID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerID="b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58" exitCode=2
Jul 08 16:54:34.874011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.873605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" event={"ID":"8240b020-722e-4446-9dc9-666e3e2b2bab","Type":"ContainerDied","Data":"b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58"}
Jul 08 16:54:34.972287 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.972245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:34.974589 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:34.974569    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls\") pod \"error-404-isvc-3816e-predictor-65dc76b695-ckgw9\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") " pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:35.211910 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.211874    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:35.342925 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.342714    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"]
Jul 08 16:54:35.345825 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:54:35.345798    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe92b54_88f7_44a6_aa95_a0d49fbf63d5.slice/crio-0903e31abd7d9ddcda06581a33223f07842e400073440e939064d851aef8b89e WatchSource:0}: Error finding container 0903e31abd7d9ddcda06581a33223f07842e400073440e939064d851aef8b89e: Status 404 returned error can't find the container with id 0903e31abd7d9ddcda06581a33223f07842e400073440e939064d851aef8b89e
Jul 08 16:54:35.879340 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.879255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" event={"ID":"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5","Type":"ContainerStarted","Data":"14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2"}
Jul 08 16:54:35.879340 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.879315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" event={"ID":"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5","Type":"ContainerStarted","Data":"7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2"}
Jul 08 16:54:35.879777 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.879346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:35.879777 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.879481    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:35.879777 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.879532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" event={"ID":"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5","Type":"ContainerStarted","Data":"0903e31abd7d9ddcda06581a33223f07842e400073440e939064d851aef8b89e"}
Jul 08 16:54:35.880796 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.880771    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:54:35.900126 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:35.900078    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podStartSLOduration=1.900063792 podStartE2EDuration="1.900063792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:54:35.898780156 +0000 UTC m=+1267.837381425" watchObservedRunningTime="2026-07-08 16:54:35.900063792 +0000 UTC m=+1267.838665062"
Jul 08 16:54:36.883865 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:36.883825    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:54:37.522165 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.522138    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:54:37.588918 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.588884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht4xj\" (UniqueName: \"kubernetes.io/projected/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-api-access-ht4xj\") pod \"8240b020-722e-4446-9dc9-666e3e2b2bab\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") "
Jul 08 16:54:37.589093 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.588939    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8240b020-722e-4446-9dc9-666e3e2b2bab-proxy-tls\") pod \"8240b020-722e-4446-9dc9-666e3e2b2bab\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") "
Jul 08 16:54:37.589093 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.589038    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-rbac-proxy-sar-config\") pod \"8240b020-722e-4446-9dc9-666e3e2b2bab\" (UID: \"8240b020-722e-4446-9dc9-666e3e2b2bab\") "
Jul 08 16:54:37.589467 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.589440    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8240b020-722e-4446-9dc9-666e3e2b2bab" (UID: "8240b020-722e-4446-9dc9-666e3e2b2bab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:54:37.591434 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.591408    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8240b020-722e-4446-9dc9-666e3e2b2bab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8240b020-722e-4446-9dc9-666e3e2b2bab" (UID: "8240b020-722e-4446-9dc9-666e3e2b2bab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:54:37.591434 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.591418    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-api-access-ht4xj" (OuterVolumeSpecName: "kube-api-access-ht4xj") pod "8240b020-722e-4446-9dc9-666e3e2b2bab" (UID: "8240b020-722e-4446-9dc9-666e3e2b2bab"). InnerVolumeSpecName "kube-api-access-ht4xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:54:37.689977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.689944    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4xj\" (UniqueName: \"kubernetes.io/projected/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-api-access-ht4xj\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:54:37.689977 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.689977    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8240b020-722e-4446-9dc9-666e3e2b2bab-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:54:37.690189 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.689990    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8240b020-722e-4446-9dc9-666e3e2b2bab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:54:37.890214 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.890180    2561 generic.go:356] "Generic (PLEG): container finished" podID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerID="9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee" exitCode=0
Jul 08 16:54:37.890703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.890268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" event={"ID":"8240b020-722e-4446-9dc9-666e3e2b2bab","Type":"ContainerDied","Data":"9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee"}
Jul 08 16:54:37.890703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.890284    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"
Jul 08 16:54:37.890703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.890318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6" event={"ID":"8240b020-722e-4446-9dc9-666e3e2b2bab","Type":"ContainerDied","Data":"23dc877bc5c282b88b11d65012f481398d8253e3f745c5f959e72f53e5005428"}
Jul 08 16:54:37.890703 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.890344    2561 scope.go:117] "RemoveContainer" containerID="b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58"
Jul 08 16:54:37.899355 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.899331    2561 scope.go:117] "RemoveContainer" containerID="9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee"
Jul 08 16:54:37.907588 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.907571    2561 scope.go:117] "RemoveContainer" containerID="b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58"
Jul 08 16:54:37.907901 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:54:37.907880    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58\": container with ID starting with b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58 not found: ID does not exist" containerID="b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58"
Jul 08 16:54:37.907998 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.907910    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58"} err="failed to get container status \"b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58\": rpc error: code = NotFound desc = could not find container \"b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58\": container with ID starting with b37eafcc7d9e0ee7c8bb5eef7c907352d6c344a686d9fd6be56f4d5775192d58 not found: ID does not exist"
Jul 08 16:54:37.907998 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.907932    2561 scope.go:117] "RemoveContainer" containerID="9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee"
Jul 08 16:54:37.908225 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:54:37.908203    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee\": container with ID starting with 9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee not found: ID does not exist" containerID="9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee"
Jul 08 16:54:37.908287 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.908236    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee"} err="failed to get container status \"9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee\": rpc error: code = NotFound desc = could not find container \"9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee\": container with ID starting with 9515133c7f1ecbf7d4f1544129e3ba154f54f71997821b25d2bd80f3a0e583ee not found: ID does not exist"
Jul 08 16:54:37.920311 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.920284    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"]
Jul 08 16:54:37.928102 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:37.928072    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fabf2-predictor-54645d4667-9pmq6"]
Jul 08 16:54:38.801275 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:38.801246    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" path="/var/lib/kubelet/pods/8240b020-722e-4446-9dc9-666e3e2b2bab/volumes"
Jul 08 16:54:41.888194 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:41.888157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:54:41.888794 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:41.888765    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:54:43.737306 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:43.737277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:54:51.888859 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:54:51.888818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:55:01.889252 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:01.889205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:55:04.665173 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.665141    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"]
Jul 08 16:55:04.665536 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.665375    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container" containerID="cri-o://6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d" gracePeriod=30
Jul 08 16:55:04.665536 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.665403    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kube-rbac-proxy" containerID="cri-o://24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408" gracePeriod=30
Jul 08 16:55:04.803278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803246    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"]
Jul 08 16:55:04.803561 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803549    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container"
Jul 08 16:55:04.803600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803564    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container"
Jul 08 16:55:04.803600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803584    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kube-rbac-proxy"
Jul 08 16:55:04.803600 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803590    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kube-rbac-proxy"
Jul 08 16:55:04.803760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803648    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kserve-container"
Jul 08 16:55:04.803760 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.803656    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8240b020-722e-4446-9dc9-666e3e2b2bab" containerName="kube-rbac-proxy"
Jul 08 16:55:04.806281 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.806265    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:04.809883 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.809860    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f879-kube-rbac-proxy-sar-config\""
Jul 08 16:55:04.810011 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.809951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f879-predictor-serving-cert\""
Jul 08 16:55:04.815608 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.815582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"]
Jul 08 16:55:04.905586 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.905551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb734e6a-0f34-4b49-a994-9fbdda88f368-proxy-tls\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:04.905781 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.905624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:04.905781 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.905668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfdb\" (UniqueName: \"kubernetes.io/projected/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-api-access-klfdb\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:04.999202 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.999125    2561 generic.go:356] "Generic (PLEG): container finished" podID="2979b4f3-c547-441d-9927-921676de4a1d" containerID="24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408" exitCode=2
Jul 08 16:55:04.999202 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:04.999183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" event={"ID":"2979b4f3-c547-441d-9927-921676de4a1d","Type":"ContainerDied","Data":"24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408"}
Jul 08 16:55:05.006513 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.006487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb734e6a-0f34-4b49-a994-9fbdda88f368-proxy-tls\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.006648 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.006524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.006648 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.006562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfdb\" (UniqueName: \"kubernetes.io/projected/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-api-access-klfdb\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.007239 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.007220    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.008872 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.008852    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb734e6a-0f34-4b49-a994-9fbdda88f368-proxy-tls\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.015500 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.015475    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfdb\" (UniqueName: \"kubernetes.io/projected/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-api-access-klfdb\") pod \"error-404-isvc-1f879-predictor-5d845d546c-bwfkj\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") " pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.117205 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.117161    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:05.248841 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:05.248810    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"]
Jul 08 16:55:05.251327 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:55:05.251267    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb734e6a_0f34_4b49_a994_9fbdda88f368.slice/crio-8fbe98f013d3a36fce44bf1647e5e5b9f908fd5e2a12b8b34c36d0152b656014 WatchSource:0}: Error finding container 8fbe98f013d3a36fce44bf1647e5e5b9f908fd5e2a12b8b34c36d0152b656014: Status 404 returned error can't find the container with id 8fbe98f013d3a36fce44bf1647e5e5b9f908fd5e2a12b8b34c36d0152b656014
Jul 08 16:55:06.005442 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.005412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" event={"ID":"bb734e6a-0f34-4b49-a994-9fbdda88f368","Type":"ContainerStarted","Data":"e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c"}
Jul 08 16:55:06.006003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.005469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:06.006003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.005491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" event={"ID":"bb734e6a-0f34-4b49-a994-9fbdda88f368","Type":"ContainerStarted","Data":"c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5"}
Jul 08 16:55:06.006003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.005516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" event={"ID":"bb734e6a-0f34-4b49-a994-9fbdda88f368","Type":"ContainerStarted","Data":"8fbe98f013d3a36fce44bf1647e5e5b9f908fd5e2a12b8b34c36d0152b656014"}
Jul 08 16:55:06.006003 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.005537    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:06.006928 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.006903    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:55:06.026759 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:06.026706    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podStartSLOduration=2.026667397 podStartE2EDuration="2.026667397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:06.026214223 +0000 UTC m=+1297.964815496" watchObservedRunningTime="2026-07-08 16:55:06.026667397 +0000 UTC m=+1297.965268665"
Jul 08 16:55:07.009930 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:07.009890    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:55:08.204663 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.204629    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:55:08.335062 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.334973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swpc2\" (UniqueName: \"kubernetes.io/projected/2979b4f3-c547-441d-9927-921676de4a1d-kube-api-access-swpc2\") pod \"2979b4f3-c547-441d-9927-921676de4a1d\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") "
Jul 08 16:55:08.335062 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.335015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2979b4f3-c547-441d-9927-921676de4a1d-kube-rbac-proxy-sar-config\") pod \"2979b4f3-c547-441d-9927-921676de4a1d\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") "
Jul 08 16:55:08.335062 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.335063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls\") pod \"2979b4f3-c547-441d-9927-921676de4a1d\" (UID: \"2979b4f3-c547-441d-9927-921676de4a1d\") "
Jul 08 16:55:08.335407 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.335375    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2979b4f3-c547-441d-9927-921676de4a1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2979b4f3-c547-441d-9927-921676de4a1d" (UID: "2979b4f3-c547-441d-9927-921676de4a1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:08.337292 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.337264    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2979b4f3-c547-441d-9927-921676de4a1d-kube-api-access-swpc2" (OuterVolumeSpecName: "kube-api-access-swpc2") pod "2979b4f3-c547-441d-9927-921676de4a1d" (UID: "2979b4f3-c547-441d-9927-921676de4a1d"). InnerVolumeSpecName "kube-api-access-swpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:08.337380 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.337349    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2979b4f3-c547-441d-9927-921676de4a1d" (UID: "2979b4f3-c547-441d-9927-921676de4a1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:08.435932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.435893    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2979b4f3-c547-441d-9927-921676de4a1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:55:08.435932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.435923    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2979b4f3-c547-441d-9927-921676de4a1d-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:55:08.435932 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:08.435934    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swpc2\" (UniqueName: \"kubernetes.io/projected/2979b4f3-c547-441d-9927-921676de4a1d-kube-api-access-swpc2\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:55:09.021392 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.021358    2561 generic.go:356] "Generic (PLEG): container finished" podID="2979b4f3-c547-441d-9927-921676de4a1d" containerID="6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d" exitCode=0
Jul 08 16:55:09.021569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.021399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" event={"ID":"2979b4f3-c547-441d-9927-921676de4a1d","Type":"ContainerDied","Data":"6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d"}
Jul 08 16:55:09.021569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.021407    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"
Jul 08 16:55:09.021569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.021453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr" event={"ID":"2979b4f3-c547-441d-9927-921676de4a1d","Type":"ContainerDied","Data":"bafc6fa38503b7b4bd51c7bb846ffaf7fc8c7903ec69d417ac72ceef826b91b5"}
Jul 08 16:55:09.021569 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.021481    2561 scope.go:117] "RemoveContainer" containerID="24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408"
Jul 08 16:55:09.029470 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.029451    2561 scope.go:117] "RemoveContainer" containerID="6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d"
Jul 08 16:55:09.037528 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.037508    2561 scope.go:117] "RemoveContainer" containerID="24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408"
Jul 08 16:55:09.037840 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:55:09.037813    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408\": container with ID starting with 24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408 not found: ID does not exist" containerID="24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408"
Jul 08 16:55:09.037940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.037851    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408"} err="failed to get container status \"24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408\": rpc error: code = NotFound desc = could not find container \"24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408\": container with ID starting with 24edac81cf28f5c117d6865b9c2b28880d9dcb030fbbcddfd3640f88d6b22408 not found: ID does not exist"
Jul 08 16:55:09.037940 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.037875    2561 scope.go:117] "RemoveContainer" containerID="6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d"
Jul 08 16:55:09.038258 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:55:09.038190    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d\": container with ID starting with 6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d not found: ID does not exist" containerID="6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d"
Jul 08 16:55:09.038258 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.038224    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d"} err="failed to get container status \"6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d\": rpc error: code = NotFound desc = could not find container \"6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d\": container with ID starting with 6c1ed311723c1ba43630124bcfea5c18e4d5784a6931bff0b601bccd745cb99d not found: ID does not exist"
Jul 08 16:55:09.040280 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.040260    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"]
Jul 08 16:55:09.043802 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:09.043783    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8befa-predictor-5957789c65-2q4tr"]
Jul 08 16:55:10.802053 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:10.802012    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2979b4f3-c547-441d-9927-921676de4a1d" path="/var/lib/kubelet/pods/2979b4f3-c547-441d-9927-921676de4a1d/volumes"
Jul 08 16:55:11.889577 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:11.889528    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 16:55:12.015196 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:12.015166    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:12.015626 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:12.015593    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:55:21.889832 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:21.889803    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:55:22.015765 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:22.015723    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:55:32.015588 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:32.015549    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:55:42.015805 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:42.015762    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 16:55:44.456300 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.456259    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"]
Jul 08 16:55:44.456756 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.456520    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container" containerID="cri-o://7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2" gracePeriod=30
Jul 08 16:55:44.456756 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.456564    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kube-rbac-proxy" containerID="cri-o://14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2" gracePeriod=30
Jul 08 16:55:44.591898 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.591866    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"]
Jul 08 16:55:44.592222 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.592209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container"
Jul 08 16:55:44.592259 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.592224    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container"
Jul 08 16:55:44.592259 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.592242    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kube-rbac-proxy"
Jul 08 16:55:44.592259 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.592248    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kube-rbac-proxy"
Jul 08 16:55:44.592349 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.592301    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kube-rbac-proxy"
Jul 08 16:55:44.592349 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.592309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2979b4f3-c547-441d-9927-921676de4a1d" containerName="kserve-container"
Jul 08 16:55:44.595229 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.595209    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.598083 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.598063    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f6dd6-predictor-serving-cert\""
Jul 08 16:55:44.598195 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.598065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f6dd6-kube-rbac-proxy-sar-config\""
Jul 08 16:55:44.605614 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.605594    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"]
Jul 08 16:55:44.632160 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.632128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqs6\" (UniqueName: \"kubernetes.io/projected/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-api-access-mnqs6\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.632306 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.632208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.632306 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.632250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.732923 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.732824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqs6\" (UniqueName: \"kubernetes.io/projected/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-api-access-mnqs6\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.732923 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.732888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.732923 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.732919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.733215 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:55:44.733040    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-serving-cert: secret "error-404-isvc-f6dd6-predictor-serving-cert" not found
Jul 08 16:55:44.733215 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:55:44.733107    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls podName:28c51aa0-a630-45c2-9cdd-4fa2361d373c nodeName:}" failed. No retries permitted until 2026-07-08 16:55:45.233092035 +0000 UTC m=+1337.171693286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls") pod "error-404-isvc-f6dd6-predictor-9b494c85-f5c49" (UID: "28c51aa0-a630-45c2-9cdd-4fa2361d373c") : secret "error-404-isvc-f6dd6-predictor-serving-cert" not found
Jul 08 16:55:44.733618 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.733595    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:44.741757 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:44.741730    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqs6\" (UniqueName: \"kubernetes.io/projected/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-api-access-mnqs6\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:45.170984 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:45.170948    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerID="14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2" exitCode=2
Jul 08 16:55:45.171153 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:45.171021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" event={"ID":"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5","Type":"ContainerDied","Data":"14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2"}
Jul 08 16:55:45.236819 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:45.236784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:45.239261 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:45.239240    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls\") pod \"error-404-isvc-f6dd6-predictor-9b494c85-f5c49\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") " pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:45.505440 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:45.505355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:45.638880 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:45.638851    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"]
Jul 08 16:55:45.640771 ip-10-0-138-233 kubenswrapper[2561]: W0708 16:55:45.640747    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c51aa0_a630_45c2_9cdd_4fa2361d373c.slice/crio-3988fe7a3c89424f4bfe3423755e8e447fd0ee73a63c2fde3a90c57e6f42c40d WatchSource:0}: Error finding container 3988fe7a3c89424f4bfe3423755e8e447fd0ee73a63c2fde3a90c57e6f42c40d: Status 404 returned error can't find the container with id 3988fe7a3c89424f4bfe3423755e8e447fd0ee73a63c2fde3a90c57e6f42c40d
Jul 08 16:55:46.177174 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:46.177142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" event={"ID":"28c51aa0-a630-45c2-9cdd-4fa2361d373c","Type":"ContainerStarted","Data":"98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970"}
Jul 08 16:55:46.177352 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:46.177206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" event={"ID":"28c51aa0-a630-45c2-9cdd-4fa2361d373c","Type":"ContainerStarted","Data":"413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19"}
Jul 08 16:55:46.177352 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:46.177233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" event={"ID":"28c51aa0-a630-45c2-9cdd-4fa2361d373c","Type":"ContainerStarted","Data":"3988fe7a3c89424f4bfe3423755e8e447fd0ee73a63c2fde3a90c57e6f42c40d"}
Jul 08 16:55:46.177352 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:46.177257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:46.196354 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:46.196304    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podStartSLOduration=2.196287904 podStartE2EDuration="2.196287904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:46.194844827 +0000 UTC m=+1338.133446094" watchObservedRunningTime="2026-07-08 16:55:46.196287904 +0000 UTC m=+1338.134889233"
Jul 08 16:55:46.884647 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:46.884604    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 08 16:55:47.180661 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.180635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:47.181877 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.181850    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 16:55:47.812420 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.812391    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:55:47.857515 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.857484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls\") pod \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") "
Jul 08 16:55:47.857708 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.857525    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnbd4\" (UniqueName: \"kubernetes.io/projected/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-api-access-lnbd4\") pod \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") "
Jul 08 16:55:47.857708 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.857573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-rbac-proxy-sar-config\") pod \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\" (UID: \"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5\") "
Jul 08 16:55:47.858022 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.857996    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" (UID: "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:55:47.859822 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.859797    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" (UID: "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:47.860586 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.860554    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-api-access-lnbd4" (OuterVolumeSpecName: "kube-api-access-lnbd4") pod "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" (UID: "bfe92b54-88f7-44a6-aa95-a0d49fbf63d5"). InnerVolumeSpecName "kube-api-access-lnbd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:47.958715 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.958606    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:55:47.958715 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.958637    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:55:47.958715 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:47.958651    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnbd4\" (UniqueName: \"kubernetes.io/projected/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5-kube-api-access-lnbd4\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 16:55:48.186278 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.186243    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerID="7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2" exitCode=0
Jul 08 16:55:48.186455 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.186284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" event={"ID":"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5","Type":"ContainerDied","Data":"7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2"}
Jul 08 16:55:48.186455 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.186351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9" event={"ID":"bfe92b54-88f7-44a6-aa95-a0d49fbf63d5","Type":"ContainerDied","Data":"0903e31abd7d9ddcda06581a33223f07842e400073440e939064d851aef8b89e"}
Jul 08 16:55:48.186455 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.186299    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"
Jul 08 16:55:48.186616 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.186380    2561 scope.go:117] "RemoveContainer" containerID="14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2"
Jul 08 16:55:48.186667 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.186627    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 16:55:48.194827 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.194810    2561 scope.go:117] "RemoveContainer" containerID="7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2"
Jul 08 16:55:48.202030 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.202011    2561 scope.go:117] "RemoveContainer" containerID="14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2"
Jul 08 16:55:48.202276 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:55:48.202257    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2\": container with ID starting with 14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2 not found: ID does not exist" containerID="14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2"
Jul 08 16:55:48.202330 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.202285    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2"} err="failed to get container status \"14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2\": rpc error: code = NotFound desc = could not find container \"14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2\": container with ID starting with 14971d8742d589a49894207d22e68038561b1699e7e99dba1c3fe88fddecc0d2 not found: ID does not exist"
Jul 08 16:55:48.202330 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.202301    2561 scope.go:117] "RemoveContainer" containerID="7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2"
Jul 08 16:55:48.202552 ip-10-0-138-233 kubenswrapper[2561]: E0708 16:55:48.202530    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2\": container with ID starting with 7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2 not found: ID does not exist" containerID="7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2"
Jul 08 16:55:48.202606 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.202562    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2"} err="failed to get container status \"7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2\": rpc error: code = NotFound desc = could not find container \"7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2\": container with ID starting with 7fc07711895f8516b8510743306f7fe5781fb7db3a71138f7d8902d70c1d68a2 not found: ID does not exist"
Jul 08 16:55:48.209799 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.209747    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"]
Jul 08 16:55:48.214708 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.214674    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3816e-predictor-65dc76b695-ckgw9"]
Jul 08 16:55:48.801702 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:48.801656    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" path="/var/lib/kubelet/pods/bfe92b54-88f7-44a6-aa95-a0d49fbf63d5/volumes"
Jul 08 16:55:52.015832 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:52.015795    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 16:55:53.192898 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:53.192865    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 16:55:53.193466 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:55:53.193437    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 16:56:03.193661 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:56:03.193621    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 16:56:13.194295 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:56:13.194249    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 16:56:23.193791 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:56:23.193750    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 16:56:33.194516 ip-10-0-138-233 kubenswrapper[2561]: I0708 16:56:33.194488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 17:04:19.572892 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.572852    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"]
Jul 08 17:04:19.573419 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.573200    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" containerID="cri-o://c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5" gracePeriod=30
Jul 08 17:04:19.573419 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.573259    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kube-rbac-proxy" containerID="cri-o://e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c" gracePeriod=30
Jul 08 17:04:19.600961 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.600937    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"]
Jul 08 17:04:19.601272 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.601259    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kube-rbac-proxy"
Jul 08 17:04:19.601272 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.601273    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kube-rbac-proxy"
Jul 08 17:04:19.601361 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.601282    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container"
Jul 08 17:04:19.601361 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.601288    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container"
Jul 08 17:04:19.601361 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.601335    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kserve-container"
Jul 08 17:04:19.601361 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.601343    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe92b54-88f7-44a6-aa95-a0d49fbf63d5" containerName="kube-rbac-proxy"
Jul 08 17:04:19.604574 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.604556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.607530 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.607455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78c4a-kube-rbac-proxy-sar-config\""
Jul 08 17:04:19.607530 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.607482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78c4a-predictor-serving-cert\""
Jul 08 17:04:19.613636 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.613615    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"]
Jul 08 17:04:19.638797 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.638766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceed8830-50d6-4bc2-81c7-eade2692b180-proxy-tls\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.638932 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.638819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jsbd\" (UniqueName: \"kubernetes.io/projected/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-api-access-6jsbd\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.638932 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.638901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.739330 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.739289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.739330 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.739336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceed8830-50d6-4bc2-81c7-eade2692b180-proxy-tls\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.739565 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.739374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jsbd\" (UniqueName: \"kubernetes.io/projected/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-api-access-6jsbd\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.739971 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.739951    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.741659 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.741636    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceed8830-50d6-4bc2-81c7-eade2692b180-proxy-tls\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.747862 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.747841    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jsbd\" (UniqueName: \"kubernetes.io/projected/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-api-access-6jsbd\") pod \"error-404-isvc-78c4a-predictor-7c46b556f9-mhct7\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") " pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:19.916014 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:19.915983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:20.045280 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.045258    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"]
Jul 08 17:04:20.047496 ip-10-0-138-233 kubenswrapper[2561]: W0708 17:04:20.047472    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceed8830_50d6_4bc2_81c7_eade2692b180.slice/crio-3c0d15444ce4453ce0dfab870d6597abeafe47428fade01af3d3594bfb556f0d WatchSource:0}: Error finding container 3c0d15444ce4453ce0dfab870d6597abeafe47428fade01af3d3594bfb556f0d: Status 404 returned error can't find the container with id 3c0d15444ce4453ce0dfab870d6597abeafe47428fade01af3d3594bfb556f0d
Jul 08 17:04:20.200785 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.200758    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerID="e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c" exitCode=2
Jul 08 17:04:20.200911 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.200832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" event={"ID":"bb734e6a-0f34-4b49-a994-9fbdda88f368","Type":"ContainerDied","Data":"e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c"}
Jul 08 17:04:20.202537 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.202516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" event={"ID":"ceed8830-50d6-4bc2-81c7-eade2692b180","Type":"ContainerStarted","Data":"ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4"}
Jul 08 17:04:20.202641 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.202560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" event={"ID":"ceed8830-50d6-4bc2-81c7-eade2692b180","Type":"ContainerStarted","Data":"851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3"}
Jul 08 17:04:20.202641 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.202586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:20.202641 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.202605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" event={"ID":"ceed8830-50d6-4bc2-81c7-eade2692b180","Type":"ContainerStarted","Data":"3c0d15444ce4453ce0dfab870d6597abeafe47428fade01af3d3594bfb556f0d"}
Jul 08 17:04:20.202641 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.202634    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:20.203901 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.203879    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 17:04:20.222170 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:20.222131    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podStartSLOduration=1.222118498 podStartE2EDuration="1.222118498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:04:20.220026914 +0000 UTC m=+1852.158628194" watchObservedRunningTime="2026-07-08 17:04:20.222118498 +0000 UTC m=+1852.160719767"
Jul 08 17:04:21.206770 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:21.206727    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 17:04:22.010139 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.010085    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 08 17:04:22.016505 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.016475    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 17:04:22.710877 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.710855    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 17:04:22.767915 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.767883    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb734e6a-0f34-4b49-a994-9fbdda88f368-proxy-tls\") pod \"bb734e6a-0f34-4b49-a994-9fbdda88f368\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") "
Jul 08 17:04:22.768067 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.767927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klfdb\" (UniqueName: \"kubernetes.io/projected/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-api-access-klfdb\") pod \"bb734e6a-0f34-4b49-a994-9fbdda88f368\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") "
Jul 08 17:04:22.768067 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.767949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-rbac-proxy-sar-config\") pod \"bb734e6a-0f34-4b49-a994-9fbdda88f368\" (UID: \"bb734e6a-0f34-4b49-a994-9fbdda88f368\") "
Jul 08 17:04:22.768343 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.768311    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb734e6a-0f34-4b49-a994-9fbdda88f368" (UID: "bb734e6a-0f34-4b49-a994-9fbdda88f368"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:04:22.769949 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.769924    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-api-access-klfdb" (OuterVolumeSpecName: "kube-api-access-klfdb") pod "bb734e6a-0f34-4b49-a994-9fbdda88f368" (UID: "bb734e6a-0f34-4b49-a994-9fbdda88f368"). InnerVolumeSpecName "kube-api-access-klfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:04:22.770043 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.770002    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb734e6a-0f34-4b49-a994-9fbdda88f368-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb734e6a-0f34-4b49-a994-9fbdda88f368" (UID: "bb734e6a-0f34-4b49-a994-9fbdda88f368"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:04:22.868749 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.868717    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb734e6a-0f34-4b49-a994-9fbdda88f368-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:04:22.868749 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.868746    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfdb\" (UniqueName: \"kubernetes.io/projected/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-api-access-klfdb\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:04:22.868926 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:22.868761    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb734e6a-0f34-4b49-a994-9fbdda88f368-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:04:23.220916 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.220875    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerID="c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5" exitCode=0
Jul 08 17:04:23.221078 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.220941    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"
Jul 08 17:04:23.221078 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.220959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" event={"ID":"bb734e6a-0f34-4b49-a994-9fbdda88f368","Type":"ContainerDied","Data":"c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5"}
Jul 08 17:04:23.221078 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.221045    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj" event={"ID":"bb734e6a-0f34-4b49-a994-9fbdda88f368","Type":"ContainerDied","Data":"8fbe98f013d3a36fce44bf1647e5e5b9f908fd5e2a12b8b34c36d0152b656014"}
Jul 08 17:04:23.221078 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.221073    2561 scope.go:117] "RemoveContainer" containerID="e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c"
Jul 08 17:04:23.228977 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.228958    2561 scope.go:117] "RemoveContainer" containerID="c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5"
Jul 08 17:04:23.236232 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.236210    2561 scope.go:117] "RemoveContainer" containerID="e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c"
Jul 08 17:04:23.236485 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:04:23.236462    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c\": container with ID starting with e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c not found: ID does not exist" containerID="e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c"
Jul 08 17:04:23.236547 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.236496    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c"} err="failed to get container status \"e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c\": rpc error: code = NotFound desc = could not find container \"e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c\": container with ID starting with e9261a552888635916cc36c09e1301c4418ff0b8f3afa11aa2e21e8ce1a7df5c not found: ID does not exist"
Jul 08 17:04:23.236547 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.236513    2561 scope.go:117] "RemoveContainer" containerID="c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5"
Jul 08 17:04:23.236739 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:04:23.236723    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5\": container with ID starting with c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5 not found: ID does not exist" containerID="c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5"
Jul 08 17:04:23.236813 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.236740    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5"} err="failed to get container status \"c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5\": rpc error: code = NotFound desc = could not find container \"c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5\": container with ID starting with c85075fe4f7c0cc84de909d7f0339734ef8d4fc65de2377bdc7d9caeb0b8d5a5 not found: ID does not exist"
Jul 08 17:04:23.241037 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.241017    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"]
Jul 08 17:04:23.247664 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:23.247642    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f879-predictor-5d845d546c-bwfkj"]
Jul 08 17:04:24.800882 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:24.800850    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" path="/var/lib/kubelet/pods/bb734e6a-0f34-4b49-a994-9fbdda88f368/volumes"
Jul 08 17:04:26.211773 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:26.211747    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:04:26.212382 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:26.212355    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 17:04:36.212622 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:36.212580    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 17:04:46.212483 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:46.212438    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 17:04:56.212521 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:56.212437    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 08 17:04:59.444032 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.443994    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"]
Jul 08 17:04:59.444528 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.444264    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container" containerID="cri-o://413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19" gracePeriod=30
Jul 08 17:04:59.444528 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.444316    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kube-rbac-proxy" containerID="cri-o://98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970" gracePeriod=30
Jul 08 17:04:59.566823 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.566791    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"]
Jul 08 17:04:59.567118 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.567106    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container"
Jul 08 17:04:59.567159 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.567120    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container"
Jul 08 17:04:59.567159 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.567140    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kube-rbac-proxy"
Jul 08 17:04:59.567159 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.567145    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kube-rbac-proxy"
Jul 08 17:04:59.567252 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.567193    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kserve-container"
Jul 08 17:04:59.567252 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.567202    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb734e6a-0f34-4b49-a994-9fbdda88f368" containerName="kube-rbac-proxy"
Jul 08 17:04:59.570056 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.570036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.572433 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.572410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-73f94-predictor-serving-cert\""
Jul 08 17:04:59.572605 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.572586    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-73f94-kube-rbac-proxy-sar-config\""
Jul 08 17:04:59.578814 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.578794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"]
Jul 08 17:04:59.671891 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.671851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.672065 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.671927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqz49\" (UniqueName: \"kubernetes.io/projected/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-api-access-fqz49\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.672065 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.671964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-proxy-tls\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.773383 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.773296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqz49\" (UniqueName: \"kubernetes.io/projected/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-api-access-fqz49\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.773383 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.773345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-proxy-tls\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.773607 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.773387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.774060 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.774040    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.775872 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.775851    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-proxy-tls\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.781928 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.781901    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqz49\" (UniqueName: \"kubernetes.io/projected/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-api-access-fqz49\") pod \"error-404-isvc-73f94-predictor-696488f899-klrs9\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") " pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:04:59.880697 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:04:59.880654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:05:00.014308 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.014260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"]
Jul 08 17:05:00.017126 ip-10-0-138-233 kubenswrapper[2561]: W0708 17:05:00.017098    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa1c3be_fd8e_44d5_8f68_d3e1fc37c399.slice/crio-1f8874ce443a065e47f0dc862123a74daf11a160134e522161dcf70f075c5522 WatchSource:0}: Error finding container 1f8874ce443a065e47f0dc862123a74daf11a160134e522161dcf70f075c5522: Status 404 returned error can't find the container with id 1f8874ce443a065e47f0dc862123a74daf11a160134e522161dcf70f075c5522
Jul 08 17:05:00.371902 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.371817    2561 generic.go:356] "Generic (PLEG): container finished" podID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerID="98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970" exitCode=2
Jul 08 17:05:00.371902 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.371887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" event={"ID":"28c51aa0-a630-45c2-9cdd-4fa2361d373c","Type":"ContainerDied","Data":"98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970"}
Jul 08 17:05:00.373670 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.373644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" event={"ID":"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399","Type":"ContainerStarted","Data":"b146271c50c274342bb65dce4363456676809a8c63765c664ece2000dde7a16c"}
Jul 08 17:05:00.373807 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.373720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:05:00.373807 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.373742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" event={"ID":"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399","Type":"ContainerStarted","Data":"1861d3d42e8c70ccde6a680e308dcb62be57427309664d4719d8dbb2e0fa12b7"}
Jul 08 17:05:00.373807 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.373761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" event={"ID":"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399","Type":"ContainerStarted","Data":"1f8874ce443a065e47f0dc862123a74daf11a160134e522161dcf70f075c5522"}
Jul 08 17:05:00.394079 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:00.394028    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podStartSLOduration=1.3940111960000001 podStartE2EDuration="1.394011196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:05:00.391717763 +0000 UTC m=+1892.330319031" watchObservedRunningTime="2026-07-08 17:05:00.394011196 +0000 UTC m=+1892.332612465"
Jul 08 17:05:01.377892 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:01.377856    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:05:01.379184 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:01.379157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 17:05:02.382584 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.382543    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 17:05:02.798351 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.798327    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 17:05:02.898044 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.898006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-rbac-proxy-sar-config\") pod \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") "
Jul 08 17:05:02.898244 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.898062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls\") pod \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") "
Jul 08 17:05:02.898244 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.898096    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnqs6\" (UniqueName: \"kubernetes.io/projected/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-api-access-mnqs6\") pod \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\" (UID: \"28c51aa0-a630-45c2-9cdd-4fa2361d373c\") "
Jul 08 17:05:02.898496 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.898461    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28c51aa0-a630-45c2-9cdd-4fa2361d373c" (UID: "28c51aa0-a630-45c2-9cdd-4fa2361d373c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:05:02.900247 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.900218    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28c51aa0-a630-45c2-9cdd-4fa2361d373c" (UID: "28c51aa0-a630-45c2-9cdd-4fa2361d373c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:05:02.900247 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.900218    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-api-access-mnqs6" (OuterVolumeSpecName: "kube-api-access-mnqs6") pod "28c51aa0-a630-45c2-9cdd-4fa2361d373c" (UID: "28c51aa0-a630-45c2-9cdd-4fa2361d373c"). InnerVolumeSpecName "kube-api-access-mnqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:05:02.999495 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.999399    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:05:02.999495 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.999433    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c51aa0-a630-45c2-9cdd-4fa2361d373c-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:05:02.999495 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:02.999443    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqs6\" (UniqueName: \"kubernetes.io/projected/28c51aa0-a630-45c2-9cdd-4fa2361d373c-kube-api-access-mnqs6\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:05:03.387900 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.387815    2561 generic.go:356] "Generic (PLEG): container finished" podID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerID="413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19" exitCode=0
Jul 08 17:05:03.387900 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.387875    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"
Jul 08 17:05:03.388321 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.387877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" event={"ID":"28c51aa0-a630-45c2-9cdd-4fa2361d373c","Type":"ContainerDied","Data":"413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19"}
Jul 08 17:05:03.388321 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.388001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49" event={"ID":"28c51aa0-a630-45c2-9cdd-4fa2361d373c","Type":"ContainerDied","Data":"3988fe7a3c89424f4bfe3423755e8e447fd0ee73a63c2fde3a90c57e6f42c40d"}
Jul 08 17:05:03.388321 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.388027    2561 scope.go:117] "RemoveContainer" containerID="98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970"
Jul 08 17:05:03.396960 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.396941    2561 scope.go:117] "RemoveContainer" containerID="413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19"
Jul 08 17:05:03.404695 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.404664    2561 scope.go:117] "RemoveContainer" containerID="98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970"
Jul 08 17:05:03.404955 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:05:03.404930    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970\": container with ID starting with 98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970 not found: ID does not exist" containerID="98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970"
Jul 08 17:05:03.405004 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.404969    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970"} err="failed to get container status \"98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970\": rpc error: code = NotFound desc = could not find container \"98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970\": container with ID starting with 98130960b9590448e96716f75d0fc838e669767c84313046432b2c85a8984970 not found: ID does not exist"
Jul 08 17:05:03.405004 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.404988    2561 scope.go:117] "RemoveContainer" containerID="413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19"
Jul 08 17:05:03.405237 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:05:03.405221    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19\": container with ID starting with 413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19 not found: ID does not exist" containerID="413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19"
Jul 08 17:05:03.405294 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.405239    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19"} err="failed to get container status \"413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19\": rpc error: code = NotFound desc = could not find container \"413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19\": container with ID starting with 413388fdd7e3092734a12b2e939b99e95e5ade5a3824fe731adfafb7774eea19 not found: ID does not exist"
Jul 08 17:05:03.412631 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.412579    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"]
Jul 08 17:05:03.414351 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:03.414327    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f6dd6-predictor-9b494c85-f5c49"]
Jul 08 17:05:04.801109 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:04.801077    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" path="/var/lib/kubelet/pods/28c51aa0-a630-45c2-9cdd-4fa2361d373c/volumes"
Jul 08 17:05:06.212852 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:06.212818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:05:07.386973 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:07.386945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:05:07.387418 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:07.387395    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 17:05:17.387386 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:17.387334    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 17:05:27.388325 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:27.388285    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 17:05:29.803726 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.803673    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"]
Jul 08 17:05:29.804151 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.803961    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container" containerID="cri-o://851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3" gracePeriod=30
Jul 08 17:05:29.804151 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.804002    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kube-rbac-proxy" containerID="cri-o://ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4" gracePeriod=30
Jul 08 17:05:29.941059 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941025    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"]
Jul 08 17:05:29.941352 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941341    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kube-rbac-proxy"
Jul 08 17:05:29.941393 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941354    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kube-rbac-proxy"
Jul 08 17:05:29.941393 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container"
Jul 08 17:05:29.941393 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941371    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container"
Jul 08 17:05:29.941479 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941421    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kube-rbac-proxy"
Jul 08 17:05:29.941479 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.941432    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c51aa0-a630-45c2-9cdd-4fa2361d373c" containerName="kserve-container"
Jul 08 17:05:29.946708 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.946521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:29.949663 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.949638    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-91fdc-predictor-serving-cert\""
Jul 08 17:05:29.949807 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.949790    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-91fdc-kube-rbac-proxy-sar-config\""
Jul 08 17:05:29.954925 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:29.954863    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"]
Jul 08 17:05:30.029424 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.029392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.029605 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.029442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5ec9e4-4350-4ed0-beca-d56c833516e7-proxy-tls\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.029605 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.029471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjllp\" (UniqueName: \"kubernetes.io/projected/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-api-access-kjllp\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.130600 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.130517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.130600 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.130565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5ec9e4-4350-4ed0-beca-d56c833516e7-proxy-tls\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.130600 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.130588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjllp\" (UniqueName: \"kubernetes.io/projected/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-api-access-kjllp\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.131224 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.131201    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.133003 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.132983    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5ec9e4-4350-4ed0-beca-d56c833516e7-proxy-tls\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.139231 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.139207    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjllp\" (UniqueName: \"kubernetes.io/projected/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-api-access-kjllp\") pod \"error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") " pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.257997 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.257954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:30.387700 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.387662    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"]
Jul 08 17:05:30.389944 ip-10-0-138-233 kubenswrapper[2561]: W0708 17:05:30.389922    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5ec9e4_4350_4ed0_beca_d56c833516e7.slice/crio-1ba39a70f9ff50a60f24300793f0cb498b75f9ddd127ac29ad3db05a0fce7034 WatchSource:0}: Error finding container 1ba39a70f9ff50a60f24300793f0cb498b75f9ddd127ac29ad3db05a0fce7034: Status 404 returned error can't find the container with id 1ba39a70f9ff50a60f24300793f0cb498b75f9ddd127ac29ad3db05a0fce7034
Jul 08 17:05:30.501867 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.501834    2561 generic.go:356] "Generic (PLEG): container finished" podID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerID="ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4" exitCode=2
Jul 08 17:05:30.502002 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.501898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" event={"ID":"ceed8830-50d6-4bc2-81c7-eade2692b180","Type":"ContainerDied","Data":"ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4"}
Jul 08 17:05:30.503383 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.503363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" event={"ID":"6a5ec9e4-4350-4ed0-beca-d56c833516e7","Type":"ContainerStarted","Data":"8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d"}
Jul 08 17:05:30.503476 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:30.503406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" event={"ID":"6a5ec9e4-4350-4ed0-beca-d56c833516e7","Type":"ContainerStarted","Data":"1ba39a70f9ff50a60f24300793f0cb498b75f9ddd127ac29ad3db05a0fce7034"}
Jul 08 17:05:31.207649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:31.207608    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 08 17:05:31.509103 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:31.509012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" event={"ID":"6a5ec9e4-4350-4ed0-beca-d56c833516e7","Type":"ContainerStarted","Data":"bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af"}
Jul 08 17:05:31.509103 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:31.509078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:31.509296 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:31.509118    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:31.510414 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:31.510393    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:05:31.526589 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:31.526544    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podStartSLOduration=2.526530035 podStartE2EDuration="2.526530035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:05:31.525588564 +0000 UTC m=+1923.464189833" watchObservedRunningTime="2026-07-08 17:05:31.526530035 +0000 UTC m=+1923.465131305"
Jul 08 17:05:32.512424 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:32.512387    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:05:32.950024 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:32.949995    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:05:33.058809 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.058727    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceed8830-50d6-4bc2-81c7-eade2692b180-proxy-tls\") pod \"ceed8830-50d6-4bc2-81c7-eade2692b180\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") "
Jul 08 17:05:33.058809 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.058772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jsbd\" (UniqueName: \"kubernetes.io/projected/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-api-access-6jsbd\") pod \"ceed8830-50d6-4bc2-81c7-eade2692b180\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") "
Jul 08 17:05:33.059019 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.058839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-rbac-proxy-sar-config\") pod \"ceed8830-50d6-4bc2-81c7-eade2692b180\" (UID: \"ceed8830-50d6-4bc2-81c7-eade2692b180\") "
Jul 08 17:05:33.059288 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.059255    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ceed8830-50d6-4bc2-81c7-eade2692b180" (UID: "ceed8830-50d6-4bc2-81c7-eade2692b180"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:05:33.060931 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.060893    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceed8830-50d6-4bc2-81c7-eade2692b180-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ceed8830-50d6-4bc2-81c7-eade2692b180" (UID: "ceed8830-50d6-4bc2-81c7-eade2692b180"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:05:33.061033 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.060962    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-api-access-6jsbd" (OuterVolumeSpecName: "kube-api-access-6jsbd") pod "ceed8830-50d6-4bc2-81c7-eade2692b180" (UID: "ceed8830-50d6-4bc2-81c7-eade2692b180"). InnerVolumeSpecName "kube-api-access-6jsbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:05:33.159670 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.159639    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ceed8830-50d6-4bc2-81c7-eade2692b180-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:05:33.159670 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.159665    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jsbd\" (UniqueName: \"kubernetes.io/projected/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-api-access-6jsbd\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:05:33.159670 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.159675    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ceed8830-50d6-4bc2-81c7-eade2692b180-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:05:33.517400 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.517368    2561 generic.go:356] "Generic (PLEG): container finished" podID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerID="851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3" exitCode=0
Jul 08 17:05:33.517884 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.517422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" event={"ID":"ceed8830-50d6-4bc2-81c7-eade2692b180","Type":"ContainerDied","Data":"851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3"}
Jul 08 17:05:33.517884 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.517426    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"
Jul 08 17:05:33.517884 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.517473    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7" event={"ID":"ceed8830-50d6-4bc2-81c7-eade2692b180","Type":"ContainerDied","Data":"3c0d15444ce4453ce0dfab870d6597abeafe47428fade01af3d3594bfb556f0d"}
Jul 08 17:05:33.517884 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.517499    2561 scope.go:117] "RemoveContainer" containerID="ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4"
Jul 08 17:05:33.525672 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.525654    2561 scope.go:117] "RemoveContainer" containerID="851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3"
Jul 08 17:05:33.532878 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.532863    2561 scope.go:117] "RemoveContainer" containerID="ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4"
Jul 08 17:05:33.533148 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:05:33.533130    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4\": container with ID starting with ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4 not found: ID does not exist" containerID="ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4"
Jul 08 17:05:33.533196 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.533157    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4"} err="failed to get container status \"ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4\": rpc error: code = NotFound desc = could not find container \"ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4\": container with ID starting with ee9827dd57d7ccb02eb54e9655e50c3bc26699f74fb87f526885975aecd5bfe4 not found: ID does not exist"
Jul 08 17:05:33.533196 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.533176    2561 scope.go:117] "RemoveContainer" containerID="851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3"
Jul 08 17:05:33.533379 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:05:33.533361    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3\": container with ID starting with 851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3 not found: ID does not exist" containerID="851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3"
Jul 08 17:05:33.533422 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.533382    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3"} err="failed to get container status \"851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3\": rpc error: code = NotFound desc = could not find container \"851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3\": container with ID starting with 851ad510c5a6a3f404c30f33c894b72bd9610b1c96eae76937014bdced5101e3 not found: ID does not exist"
Jul 08 17:05:33.538761 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.538728    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"]
Jul 08 17:05:33.541992 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:33.541974    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78c4a-predictor-7c46b556f9-mhct7"]
Jul 08 17:05:34.801643 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:34.801613    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" path="/var/lib/kubelet/pods/ceed8830-50d6-4bc2-81c7-eade2692b180/volumes"
Jul 08 17:05:37.387579 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:37.387538    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 17:05:37.517925 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:37.517894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:05:37.518506 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:37.518475    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:05:47.388208 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:47.388177    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:05:47.518852 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:47.518811    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:05:57.518424 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:05:57.518383    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:06:07.518791 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:06:07.518746    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:06:17.518841 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:06:17.518811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:14:44.708093 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:44.708055    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"]
Jul 08 17:14:44.710523 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:44.708367    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" containerID="cri-o://8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d" gracePeriod=30
Jul 08 17:14:44.710523 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:44.708367    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kube-rbac-proxy" containerID="cri-o://bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af" gracePeriod=30
Jul 08 17:14:45.638548 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:45.638517    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerID="bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af" exitCode=2
Jul 08 17:14:45.638747 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:45.638594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" event={"ID":"6a5ec9e4-4350-4ed0-beca-d56c833516e7","Type":"ContainerDied","Data":"bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af"}
Jul 08 17:14:47.513440 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.513395    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 08 17:14:47.518920 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.518894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 17:14:47.847888 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.847863    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:14:47.905962 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.905932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-rbac-proxy-sar-config\") pod \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") "
Jul 08 17:14:47.906124 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.905979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjllp\" (UniqueName: \"kubernetes.io/projected/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-api-access-kjllp\") pod \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") "
Jul 08 17:14:47.906124 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.906040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5ec9e4-4350-4ed0-beca-d56c833516e7-proxy-tls\") pod \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\" (UID: \"6a5ec9e4-4350-4ed0-beca-d56c833516e7\") "
Jul 08 17:14:47.906702 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.906349    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a5ec9e4-4350-4ed0-beca-d56c833516e7" (UID: "6a5ec9e4-4350-4ed0-beca-d56c833516e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:14:47.908176 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.908147    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5ec9e4-4350-4ed0-beca-d56c833516e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a5ec9e4-4350-4ed0-beca-d56c833516e7" (UID: "6a5ec9e4-4350-4ed0-beca-d56c833516e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:14:47.908176 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:47.908171    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-api-access-kjllp" (OuterVolumeSpecName: "kube-api-access-kjllp") pod "6a5ec9e4-4350-4ed0-beca-d56c833516e7" (UID: "6a5ec9e4-4350-4ed0-beca-d56c833516e7"). InnerVolumeSpecName "kube-api-access-kjllp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:14:48.007738 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.007702    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a5ec9e4-4350-4ed0-beca-d56c833516e7-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:14:48.007738 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.007731    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:14:48.007738 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.007741    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjllp\" (UniqueName: \"kubernetes.io/projected/6a5ec9e4-4350-4ed0-beca-d56c833516e7-kube-api-access-kjllp\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:14:48.652358 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.652329    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerID="8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d" exitCode=0
Jul 08 17:14:48.652747 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.652368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" event={"ID":"6a5ec9e4-4350-4ed0-beca-d56c833516e7","Type":"ContainerDied","Data":"8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d"}
Jul 08 17:14:48.652747 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.652375    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"
Jul 08 17:14:48.652747 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.652412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq" event={"ID":"6a5ec9e4-4350-4ed0-beca-d56c833516e7","Type":"ContainerDied","Data":"1ba39a70f9ff50a60f24300793f0cb498b75f9ddd127ac29ad3db05a0fce7034"}
Jul 08 17:14:48.652747 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.652440    2561 scope.go:117] "RemoveContainer" containerID="bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af"
Jul 08 17:14:48.661300 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.661280    2561 scope.go:117] "RemoveContainer" containerID="8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d"
Jul 08 17:14:48.672789 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.672671    2561 scope.go:117] "RemoveContainer" containerID="bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af"
Jul 08 17:14:48.673097 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:14:48.673075    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af\": container with ID starting with bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af not found: ID does not exist" containerID="bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af"
Jul 08 17:14:48.673164 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.673104    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af"} err="failed to get container status \"bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af\": rpc error: code = NotFound desc = could not find container \"bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af\": container with ID starting with bbf3d282d54b16d63d5d442ca9ec35aca6eeac653b9b31666883e337b427d4af not found: ID does not exist"
Jul 08 17:14:48.673164 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.673119    2561 scope.go:117] "RemoveContainer" containerID="8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d"
Jul 08 17:14:48.673353 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:14:48.673333    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d\": container with ID starting with 8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d not found: ID does not exist" containerID="8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d"
Jul 08 17:14:48.673448 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.673354    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d"} err="failed to get container status \"8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d\": rpc error: code = NotFound desc = could not find container \"8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d\": container with ID starting with 8e87fe28cc8c4f9e96784e79085757e7fa044d81ee7e1a3abb3633ce6fedc08d not found: ID does not exist"
Jul 08 17:14:48.680509 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.680488    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"]
Jul 08 17:14:48.686284 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.686265    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-91fdc-predictor-5d77898ff9-wkvdq"]
Jul 08 17:14:48.801224 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:14:48.801195    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" path="/var/lib/kubelet/pods/6a5ec9e4-4350-4ed0-beca-d56c833516e7/volumes"
Jul 08 17:22:18.969735 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:18.969698    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"]
Jul 08 17:22:18.972248 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:18.970032    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container" containerID="cri-o://1861d3d42e8c70ccde6a680e308dcb62be57427309664d4719d8dbb2e0fa12b7" gracePeriod=30
Jul 08 17:22:18.972248 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:18.970034    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kube-rbac-proxy" containerID="cri-o://b146271c50c274342bb65dce4363456676809a8c63765c664ece2000dde7a16c" gracePeriod=30
Jul 08 17:22:19.364151 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.364053    2561 generic.go:356] "Generic (PLEG): container finished" podID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerID="b146271c50c274342bb65dce4363456676809a8c63765c664ece2000dde7a16c" exitCode=2
Jul 08 17:22:19.364298 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.364139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" event={"ID":"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399","Type":"ContainerDied","Data":"b146271c50c274342bb65dce4363456676809a8c63765c664ece2000dde7a16c"}
Jul 08 17:22:19.511746 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.511706    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7tbrn/must-gather-4rz5c"]
Jul 08 17:22:19.512088 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512072    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512092    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512113    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512122    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512137    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kube-rbac-proxy"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512145    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kube-rbac-proxy"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512163    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kube-rbac-proxy"
Jul 08 17:22:19.512175 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512172    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kube-rbac-proxy"
Jul 08 17:22:19.512523 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512249    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kserve-container"
Jul 08 17:22:19.512523 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512262    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceed8830-50d6-4bc2-81c7-eade2692b180" containerName="kube-rbac-proxy"
Jul 08 17:22:19.512523 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512274    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kube-rbac-proxy"
Jul 08 17:22:19.512523 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.512285    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5ec9e4-4350-4ed0-beca-d56c833516e7" containerName="kserve-container"
Jul 08 17:22:19.515185 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.515165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.517967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.517947    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7tbrn\"/\"openshift-service-ca.crt\""
Jul 08 17:22:19.519187 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.519170    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7tbrn\"/\"default-dockercfg-9mvlj\""
Jul 08 17:22:19.519276 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.519169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7tbrn\"/\"kube-root-ca.crt\""
Jul 08 17:22:19.525312 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.525287    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7tbrn/must-gather-4rz5c"]
Jul 08 17:22:19.683256 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.683219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/234615db-cfb1-4cb9-bdaa-85fbe4828a36-must-gather-output\") pod \"must-gather-4rz5c\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") " pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.683440 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.683277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7bj\" (UniqueName: \"kubernetes.io/projected/234615db-cfb1-4cb9-bdaa-85fbe4828a36-kube-api-access-wl7bj\") pod \"must-gather-4rz5c\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") " pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.784547 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.784508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/234615db-cfb1-4cb9-bdaa-85fbe4828a36-must-gather-output\") pod \"must-gather-4rz5c\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") " pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.784742 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.784578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7bj\" (UniqueName: \"kubernetes.io/projected/234615db-cfb1-4cb9-bdaa-85fbe4828a36-kube-api-access-wl7bj\") pod \"must-gather-4rz5c\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") " pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.784966 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.784944    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/234615db-cfb1-4cb9-bdaa-85fbe4828a36-must-gather-output\") pod \"must-gather-4rz5c\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") " pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.793245 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.793227    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7bj\" (UniqueName: \"kubernetes.io/projected/234615db-cfb1-4cb9-bdaa-85fbe4828a36-kube-api-access-wl7bj\") pod \"must-gather-4rz5c\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") " pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.824338 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.824296    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:19.955345 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.955322    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7tbrn/must-gather-4rz5c"]
Jul 08 17:22:19.958387 ip-10-0-138-233 kubenswrapper[2561]: W0708 17:22:19.958355    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod234615db_cfb1_4cb9_bdaa_85fbe4828a36.slice/crio-d387b630ff9828aa6d8218c980c5e1d65087319480dbbdf94d9176d345cb8918 WatchSource:0}: Error finding container d387b630ff9828aa6d8218c980c5e1d65087319480dbbdf94d9176d345cb8918: Status 404 returned error can't find the container with id d387b630ff9828aa6d8218c980c5e1d65087319480dbbdf94d9176d345cb8918
Jul 08 17:22:19.960176 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:19.960157    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:22:20.369166 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:20.369078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7tbrn/must-gather-4rz5c" event={"ID":"234615db-cfb1-4cb9-bdaa-85fbe4828a36","Type":"ContainerStarted","Data":"d387b630ff9828aa6d8218c980c5e1d65087319480dbbdf94d9176d345cb8918"}
Jul 08 17:22:22.382862 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:22.382826    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 17:22:24.390115 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:24.390074    2561 generic.go:356] "Generic (PLEG): container finished" podID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerID="1861d3d42e8c70ccde6a680e308dcb62be57427309664d4719d8dbb2e0fa12b7" exitCode=0
Jul 08 17:22:24.390115 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:24.390101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" event={"ID":"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399","Type":"ContainerDied","Data":"1861d3d42e8c70ccde6a680e308dcb62be57427309664d4719d8dbb2e0fa12b7"}
Jul 08 17:22:24.958327 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:24.958299    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:22:25.035649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.035612    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-proxy-tls\") pod \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") "
Jul 08 17:22:25.035853 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.035669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqz49\" (UniqueName: \"kubernetes.io/projected/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-api-access-fqz49\") pod \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") "
Jul 08 17:22:25.035853 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.035718    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-rbac-proxy-sar-config\") pod \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\" (UID: \"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399\") "
Jul 08 17:22:25.036205 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.036173    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" (UID: "5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:22:25.038076 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.038036    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-api-access-fqz49" (OuterVolumeSpecName: "kube-api-access-fqz49") pod "5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" (UID: "5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399"). InnerVolumeSpecName "kube-api-access-fqz49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:22:25.038183 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.038108    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" (UID: "5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:22:25.137031 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.136995    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-proxy-tls\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:22:25.137031 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.137029    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqz49\" (UniqueName: \"kubernetes.io/projected/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-api-access-fqz49\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:22:25.137278 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.137048    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:22:25.396347 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.396315    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"
Jul 08 17:22:25.396347 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.396321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9" event={"ID":"5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399","Type":"ContainerDied","Data":"1f8874ce443a065e47f0dc862123a74daf11a160134e522161dcf70f075c5522"}
Jul 08 17:22:25.396995 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.396403    2561 scope.go:117] "RemoveContainer" containerID="b146271c50c274342bb65dce4363456676809a8c63765c664ece2000dde7a16c"
Jul 08 17:22:25.420337 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.420305    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"]
Jul 08 17:22:25.424071 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:25.424048    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-73f94-predictor-696488f899-klrs9"]
Jul 08 17:22:26.053806 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:26.053531    2561 scope.go:117] "RemoveContainer" containerID="1861d3d42e8c70ccde6a680e308dcb62be57427309664d4719d8dbb2e0fa12b7"
Jul 08 17:22:26.403143 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:26.403111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7tbrn/must-gather-4rz5c" event={"ID":"234615db-cfb1-4cb9-bdaa-85fbe4828a36","Type":"ContainerStarted","Data":"c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde"}
Jul 08 17:22:26.403608 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:26.403159    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7tbrn/must-gather-4rz5c" event={"ID":"234615db-cfb1-4cb9-bdaa-85fbe4828a36","Type":"ContainerStarted","Data":"7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"}
Jul 08 17:22:26.422864 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:26.422818    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7tbrn/must-gather-4rz5c" podStartSLOduration=1.26066857 podStartE2EDuration="7.422800212s" totalImagesPullingTime="6.162131642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:22:26.421980553 +0000 UTC m=+2938.360581824" watchObservedRunningTime="2026-07-08 17:22:26.422800212 +0000 UTC m=+2938.361401481"
Jul 08 17:22:26.803934 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:26.803904    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" path="/var/lib/kubelet/pods/5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399/volumes"
Jul 08 17:22:48.501902 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:48.501867    2561 generic.go:356] "Generic (PLEG): container finished" podID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerID="7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5" exitCode=0
Jul 08 17:22:48.502434 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:48.501948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7tbrn/must-gather-4rz5c" event={"ID":"234615db-cfb1-4cb9-bdaa-85fbe4828a36","Type":"ContainerDied","Data":"7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"}
Jul 08 17:22:48.502434 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:48.502349    2561 scope.go:117] "RemoveContainer" containerID="7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"
Jul 08 17:22:49.423367 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:49.423299    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7tbrn_must-gather-4rz5c_234615db-cfb1-4cb9-bdaa-85fbe4828a36/gather/0.log"
Jul 08 17:22:50.010537 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010500    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5dxj/must-gather-nntng"]
Jul 08 17:22:50.010967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010838    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container"
Jul 08 17:22:50.010967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010850    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container"
Jul 08 17:22:50.010967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010861    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kube-rbac-proxy"
Jul 08 17:22:50.010967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010867    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kube-rbac-proxy"
Jul 08 17:22:50.010967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010917    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kube-rbac-proxy"
Jul 08 17:22:50.010967 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.010927    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa1c3be-fd8e-44d5-8f68-d3e1fc37c399" containerName="kserve-container"
Jul 08 17:22:50.013084 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.013063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.015867 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.015845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5dxj\"/\"default-dockercfg-ttnqq\""
Jul 08 17:22:50.016100 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.016085    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5dxj\"/\"kube-root-ca.crt\""
Jul 08 17:22:50.016196 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.016120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5dxj\"/\"openshift-service-ca.crt\""
Jul 08 17:22:50.022350 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.022330    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5dxj/must-gather-nntng"]
Jul 08 17:22:50.149328 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.149290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccqv\" (UniqueName: \"kubernetes.io/projected/b6f95199-6ea0-4aa6-80a7-1fb7ac23b683-kube-api-access-jccqv\") pod \"must-gather-nntng\" (UID: \"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683\") " pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.149526 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.149367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6f95199-6ea0-4aa6-80a7-1fb7ac23b683-must-gather-output\") pod \"must-gather-nntng\" (UID: \"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683\") " pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.250635 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.250595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccqv\" (UniqueName: \"kubernetes.io/projected/b6f95199-6ea0-4aa6-80a7-1fb7ac23b683-kube-api-access-jccqv\") pod \"must-gather-nntng\" (UID: \"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683\") " pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.250849 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.250655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6f95199-6ea0-4aa6-80a7-1fb7ac23b683-must-gather-output\") pod \"must-gather-nntng\" (UID: \"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683\") " pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.251028 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.251011    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b6f95199-6ea0-4aa6-80a7-1fb7ac23b683-must-gather-output\") pod \"must-gather-nntng\" (UID: \"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683\") " pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.259400 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.259368    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccqv\" (UniqueName: \"kubernetes.io/projected/b6f95199-6ea0-4aa6-80a7-1fb7ac23b683-kube-api-access-jccqv\") pod \"must-gather-nntng\" (UID: \"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683\") " pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.322899 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.322826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5dxj/must-gather-nntng"
Jul 08 17:22:50.454346 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.454320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5dxj/must-gather-nntng"]
Jul 08 17:22:50.456791 ip-10-0-138-233 kubenswrapper[2561]: W0708 17:22:50.456760    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6f95199_6ea0_4aa6_80a7_1fb7ac23b683.slice/crio-d7be567d9bd66a4d7fdbdf43fdb96d745352c414fe43bb116b2fab72c8fcac80 WatchSource:0}: Error finding container d7be567d9bd66a4d7fdbdf43fdb96d745352c414fe43bb116b2fab72c8fcac80: Status 404 returned error can't find the container with id d7be567d9bd66a4d7fdbdf43fdb96d745352c414fe43bb116b2fab72c8fcac80
Jul 08 17:22:50.510274 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:50.510240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5dxj/must-gather-nntng" event={"ID":"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683","Type":"ContainerStarted","Data":"d7be567d9bd66a4d7fdbdf43fdb96d745352c414fe43bb116b2fab72c8fcac80"}
Jul 08 17:22:51.516439 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:51.516408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5dxj/must-gather-nntng" event={"ID":"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683","Type":"ContainerStarted","Data":"0f037c223c1305ae933b38a75a03a679bf6e69078380e4faee2b256c3941b936"}
Jul 08 17:22:51.516811 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:51.516460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5dxj/must-gather-nntng" event={"ID":"b6f95199-6ea0-4aa6-80a7-1fb7ac23b683","Type":"ContainerStarted","Data":"7dda497a22de9b6352975ccc5559968fe5c1e3e36cff2b98bede5bd614f6a6d1"}
Jul 08 17:22:51.533731 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:51.533656    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q5dxj/must-gather-nntng" podStartSLOduration=1.67339059 podStartE2EDuration="2.533640523s" totalImagesPullingTime="860.249933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:22:51.533062495 +0000 UTC m=+2963.471663767" watchObservedRunningTime="2026-07-08 17:22:51.533640523 +0000 UTC m=+2963.472241792"
Jul 08 17:22:52.683568 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:52.683539    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dlb5z_03e71b89-45d3-415c-844d-c81aa15d87f7/global-pull-secret-syncer/0.log"
Jul 08 17:22:52.881924 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:52.881892    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8lfxt_b19d6066-979b-4e85-9e24-6b20dc7ad427/konnectivity-agent/0.log"
Jul 08 17:22:52.952442 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:52.952365    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-233.ec2.internal_938243ce709c303fec6b51c38490eae8/haproxy/0.log"
Jul 08 17:22:54.864134 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:54.864094    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7tbrn/must-gather-4rz5c"]
Jul 08 17:22:54.864631 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:54.864375    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-7tbrn/must-gather-4rz5c" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="copy" containerID="cri-o://c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde" gracePeriod=2
Jul 08 17:22:54.869058 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:54.869035    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7tbrn/must-gather-4rz5c"]
Jul 08 17:22:54.871268 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:22:54.869873    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4rz5c\" is forbidden: User \"system:node:ip-10-0-138-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7tbrn\": no relationship found between node 'ip-10-0-138-233.ec2.internal' and this object" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:55.257712 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.255477    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7tbrn_must-gather-4rz5c_234615db-cfb1-4cb9-bdaa-85fbe4828a36/copy/0.log"
Jul 08 17:22:55.264706 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.258037    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:55.264706 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:22:55.260458    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4rz5c\" is forbidden: User \"system:node:ip-10-0-138-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7tbrn\": no relationship found between node 'ip-10-0-138-233.ec2.internal' and this object" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:55.416261 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.416072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/234615db-cfb1-4cb9-bdaa-85fbe4828a36-must-gather-output\") pod \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") "
Jul 08 17:22:55.416261 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.416133    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl7bj\" (UniqueName: \"kubernetes.io/projected/234615db-cfb1-4cb9-bdaa-85fbe4828a36-kube-api-access-wl7bj\") pod \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\" (UID: \"234615db-cfb1-4cb9-bdaa-85fbe4828a36\") "
Jul 08 17:22:55.418311 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.418229    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234615db-cfb1-4cb9-bdaa-85fbe4828a36-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "234615db-cfb1-4cb9-bdaa-85fbe4828a36" (UID: "234615db-cfb1-4cb9-bdaa-85fbe4828a36"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:22:55.420452 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.420413    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234615db-cfb1-4cb9-bdaa-85fbe4828a36-kube-api-access-wl7bj" (OuterVolumeSpecName: "kube-api-access-wl7bj") pod "234615db-cfb1-4cb9-bdaa-85fbe4828a36" (UID: "234615db-cfb1-4cb9-bdaa-85fbe4828a36"). InnerVolumeSpecName "kube-api-access-wl7bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:22:55.517140 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.517047    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/234615db-cfb1-4cb9-bdaa-85fbe4828a36-must-gather-output\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:22:55.517140 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.517090    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7bj\" (UniqueName: \"kubernetes.io/projected/234615db-cfb1-4cb9-bdaa-85fbe4828a36-kube-api-access-wl7bj\") on node \"ip-10-0-138-233.ec2.internal\" DevicePath \"\""
Jul 08 17:22:55.541430 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.541395    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7tbrn_must-gather-4rz5c_234615db-cfb1-4cb9-bdaa-85fbe4828a36/copy/0.log"
Jul 08 17:22:55.541870 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.541837    2561 generic.go:356] "Generic (PLEG): container finished" podID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerID="c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde" exitCode=143
Jul 08 17:22:55.541991 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.541976    2561 scope.go:117] "RemoveContainer" containerID="c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde"
Jul 08 17:22:55.542221 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.542203    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:55.553044 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:22:55.553007    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4rz5c\" is forbidden: User \"system:node:ip-10-0-138-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7tbrn\": no relationship found between node 'ip-10-0-138-233.ec2.internal' and this object" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:55.565215 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.564978    2561 scope.go:117] "RemoveContainer" containerID="7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"
Jul 08 17:22:55.571888 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:22:55.571844    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4rz5c\" is forbidden: User \"system:node:ip-10-0-138-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7tbrn\": no relationship found between node 'ip-10-0-138-233.ec2.internal' and this object" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" pod="openshift-must-gather-7tbrn/must-gather-4rz5c"
Jul 08 17:22:55.584646 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.584618    2561 scope.go:117] "RemoveContainer" containerID="c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde"
Jul 08 17:22:55.585240 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:22:55.585208    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde\": container with ID starting with c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde not found: ID does not exist" containerID="c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde"
Jul 08 17:22:55.585341 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.585255    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde"} err="failed to get container status \"c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde\": rpc error: code = NotFound desc = could not find container \"c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde\": container with ID starting with c44c34a782dc6c66a40cd8983b180fa3545b83fba454e16e56976a8876592fde not found: ID does not exist"
Jul 08 17:22:55.585341 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.585280    2561 scope.go:117] "RemoveContainer" containerID="7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"
Jul 08 17:22:55.585764 ip-10-0-138-233 kubenswrapper[2561]: E0708 17:22:55.585725    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5\": container with ID starting with 7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5 not found: ID does not exist" containerID="7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"
Jul 08 17:22:55.585926 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:55.585904    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5"} err="failed to get container status \"7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5\": rpc error: code = NotFound desc = could not find container \"7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5\": container with ID starting with 7f99a96059914f92279a6f43d0b387eb3125345fa73f7da824442e70f8a37eb5 not found: ID does not exist"
Jul 08 17:22:56.803914 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:56.803880    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" path="/var/lib/kubelet/pods/234615db-cfb1-4cb9-bdaa-85fbe4828a36/volumes"
Jul 08 17:22:57.136467 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.136435    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ctcsm_25608786-14b6-429f-912c-1ca46206e171/node-exporter/0.log"
Jul 08 17:22:57.163094 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.163064    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ctcsm_25608786-14b6-429f-912c-1ca46206e171/kube-rbac-proxy/0.log"
Jul 08 17:22:57.192364 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.192334    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ctcsm_25608786-14b6-429f-912c-1ca46206e171/init-textfile/0.log"
Jul 08 17:22:57.343136 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.343029    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-4zbl2_6ad20774-3ad0-4942-a467-66e8da1044fe/kube-rbac-proxy-main/0.log"
Jul 08 17:22:57.408310 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.408277    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-4zbl2_6ad20774-3ad0-4942-a467-66e8da1044fe/kube-rbac-proxy-self/0.log"
Jul 08 17:22:57.435717 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.435663    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-4zbl2_6ad20774-3ad0-4942-a467-66e8da1044fe/openshift-state-metrics/0.log"
Jul 08 17:22:57.484891 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.484832    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/prometheus/0.log"
Jul 08 17:22:57.508910 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.508862    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/config-reloader/0.log"
Jul 08 17:22:57.538222 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.538184    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/thanos-sidecar/0.log"
Jul 08 17:22:57.568227 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.568190    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/kube-rbac-proxy-web/0.log"
Jul 08 17:22:57.592058 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.592035    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/kube-rbac-proxy/0.log"
Jul 08 17:22:57.619429 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.619347    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/kube-rbac-proxy-thanos/0.log"
Jul 08 17:22:57.643170 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.643146    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2429eaff-a852-4855-9508-1a83644fa586/init-config-reloader/0.log"
Jul 08 17:22:57.677382 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.677350    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-mzbsx_dc9e4e10-677d-4ad7-a326-ede65aa451b8/prometheus-operator/0.log"
Jul 08 17:22:57.696895 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.696855    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-mzbsx_dc9e4e10-677d-4ad7-a326-ede65aa451b8/kube-rbac-proxy/0.log"
Jul 08 17:22:57.857233 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.857199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97cbc85c4-m9smd_0a08708d-7963-47ce-8f34-45de1f6337c2/thanos-query/0.log"
Jul 08 17:22:57.888940 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.888857    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97cbc85c4-m9smd_0a08708d-7963-47ce-8f34-45de1f6337c2/kube-rbac-proxy-web/0.log"
Jul 08 17:22:57.913038 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.913003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97cbc85c4-m9smd_0a08708d-7963-47ce-8f34-45de1f6337c2/kube-rbac-proxy/0.log"
Jul 08 17:22:57.940937 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.940901    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97cbc85c4-m9smd_0a08708d-7963-47ce-8f34-45de1f6337c2/prom-label-proxy/0.log"
Jul 08 17:22:57.968084 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.968044    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97cbc85c4-m9smd_0a08708d-7963-47ce-8f34-45de1f6337c2/kube-rbac-proxy-rules/0.log"
Jul 08 17:22:57.995515 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:22:57.995482    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-97cbc85c4-m9smd_0a08708d-7963-47ce-8f34-45de1f6337c2/kube-rbac-proxy-metrics/0.log"
Jul 08 17:23:01.343872 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.343807    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"]
Jul 08 17:23:01.344459 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.344361    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="gather"
Jul 08 17:23:01.344459 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.344390    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="gather"
Jul 08 17:23:01.344459 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.344418    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="copy"
Jul 08 17:23:01.344459 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.344426    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="copy"
Jul 08 17:23:01.344676 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.344505    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="gather"
Jul 08 17:23:01.344676 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.344519    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="234615db-cfb1-4cb9-bdaa-85fbe4828a36" containerName="copy"
Jul 08 17:23:01.348089 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.348064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.358393 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.358366    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"]
Jul 08 17:23:01.475921 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.475888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-podres\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.476100 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.475943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjcz\" (UniqueName: \"kubernetes.io/projected/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-kube-api-access-cvjcz\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.476100 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.476056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-lib-modules\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.476212 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.476118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-sys\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.476212 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.476158    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-proc\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577452 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-proc\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-podres\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577535    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-proc\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjcz\" (UniqueName: \"kubernetes.io/projected/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-kube-api-access-cvjcz\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-lib-modules\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577649 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-sys\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577936 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577673    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-podres\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577936 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577736    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-sys\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.577936 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.577840    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-lib-modules\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.586929 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.586901    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjcz\" (UniqueName: \"kubernetes.io/projected/3c8f34dc-753d-4bd8-ad8c-37f7593f0197-kube-api-access-cvjcz\") pod \"perf-node-gather-daemonset-t82rq\" (UID: \"3c8f34dc-753d-4bd8-ad8c-37f7593f0197\") " pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.662742 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.662429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:01.824624 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.824585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"]
Jul 08 17:23:01.984747 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:01.984658    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-czxrs_d33f300c-dab4-44c7-9341-cea20d6fbd2c/dns/0.log"
Jul 08 17:23:02.007328 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.007298    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-czxrs_d33f300c-dab4-44c7-9341-cea20d6fbd2c/kube-rbac-proxy/0.log"
Jul 08 17:23:02.161944 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.161911    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dcpxh_682cd642-9143-47ab-92ed-d33cfbd3eec9/dns-node-resolver/0.log"
Jul 08 17:23:02.585757 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.585723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq" event={"ID":"3c8f34dc-753d-4bd8-ad8c-37f7593f0197","Type":"ContainerStarted","Data":"f96dc62f222f22b05868e2a4bb1633fe9c856722c35fc8c043684dfa3a90fdde"}
Jul 08 17:23:02.586191 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.585783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq" event={"ID":"3c8f34dc-753d-4bd8-ad8c-37f7593f0197","Type":"ContainerStarted","Data":"bed05adbdf8e99a49333bce15968ad317cd6e2cba56c8103645f4d64b97428a8"}
Jul 08 17:23:02.586191 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.585825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:02.604641 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.604576    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq" podStartSLOduration=1.604539556 podStartE2EDuration="1.604539556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:23:02.603129807 +0000 UTC m=+2974.541731077" watchObservedRunningTime="2026-07-08 17:23:02.604539556 +0000 UTC m=+2974.543140828"
Jul 08 17:23:02.719916 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:02.719877    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j8zcx_22afd025-a356-4e25-a21b-b1d8cd06dc7c/node-ca/0.log"
Jul 08 17:23:03.967035 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:03.967004    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2f8x4_0207049b-d24a-4e85-a473-97a5b8587bdf/serve-healthcheck-canary/0.log"
Jul 08 17:23:04.421379 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:04.421352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j58g5_fcc2ec40-2691-48c8-88df-a6f7e67ee07c/kube-rbac-proxy/0.log"
Jul 08 17:23:04.443036 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:04.443005    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j58g5_fcc2ec40-2691-48c8-88df-a6f7e67ee07c/exporter/0.log"
Jul 08 17:23:04.464565 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:04.464539    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j58g5_fcc2ec40-2691-48c8-88df-a6f7e67ee07c/extractor/0.log"
Jul 08 17:23:06.802695 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:06.802664    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-q86mf_b15bb9e6-16e4-473f-b909-f5902f0d2aaf/manager/0.log"
Jul 08 17:23:06.833584 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:06.833558    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4ndmt_328062d5-6793-4964-971f-d4c325e3fd59/s3-init/0.log"
Jul 08 17:23:06.850227 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:06.850196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4ndmt_328062d5-6793-4964-971f-d4c325e3fd59/download-hf-model/0.log"
Jul 08 17:23:06.879851 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:06.879828    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-wfh2z_a9fe57fc-d963-420a-8ceb-25ed47e43a15/seaweedfs/0.log"
Jul 08 17:23:08.601456 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:08.601428    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5dxj/perf-node-gather-daemonset-t82rq"
Jul 08 17:23:12.768865 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.768836    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/kube-multus-additional-cni-plugins/0.log"
Jul 08 17:23:12.792106 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.792076    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/egress-router-binary-copy/0.log"
Jul 08 17:23:12.812351 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.812326    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/cni-plugins/0.log"
Jul 08 17:23:12.833006 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.832980    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/bond-cni-plugin/0.log"
Jul 08 17:23:12.854819 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.854780    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/routeoverride-cni/0.log"
Jul 08 17:23:12.875874 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.875847    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/whereabouts-cni-bincopy/0.log"
Jul 08 17:23:12.896978 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.896952    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xnrtk_9cc40af6-5278-491d-83d0-7c61cc70943c/whereabouts-cni/0.log"
Jul 08 17:23:12.927867 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:12.927838    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f6m68_075bcaa7-4039-43d5-8c2a-a654180769f9/kube-multus/0.log"
Jul 08 17:23:13.019279 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.019195    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pdp4m_b11058bb-8eed-4973-b455-a320ededc11b/network-metrics-daemon/0.log"
Jul 08 17:23:13.040597 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.040568    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pdp4m_b11058bb-8eed-4973-b455-a320ededc11b/kube-rbac-proxy/0.log"
Jul 08 17:23:13.827706 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.827658    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/ovn-controller/0.log"
Jul 08 17:23:13.851458 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.851432    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/ovn-acl-logging/0.log"
Jul 08 17:23:13.875867 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.875827    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/kube-rbac-proxy-node/0.log"
Jul 08 17:23:13.902977 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.902949    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 17:23:13.927237 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.927206    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/northd/0.log"
Jul 08 17:23:13.950787 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.950763    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/nbdb/0.log"
Jul 08 17:23:13.977480 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:13.977451    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/sbdb/0.log"
Jul 08 17:23:14.024401 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:14.024365    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6m2cv_d1b5fdf1-1f55-47e4-a91a-909df54d83bb/ovnkube-controller/0.log"
Jul 08 17:23:15.512931 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:15.512902    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ddnsp_9a6f26ab-7d25-4a7c-b527-d7a717586e35/network-check-target-container/0.log"
Jul 08 17:23:16.495538 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:16.495515    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lnd8n_ef661a59-4e98-4d7b-8a57-02b0ec567c42/iptables-alerter/0.log"
Jul 08 17:23:17.260742 ip-10-0-138-233 kubenswrapper[2561]: I0708 17:23:17.260711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bgpfl_8d5696ce-42fc-4253-b113-85471cef9d7f/tuned/0.log"