Jul 09 18:03:33.900356 ip-10-0-142-14 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:03:34.347140 ip-10-0-142-14 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:34.347140 ip-10-0-142-14 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:03:34.347140 ip-10-0-142-14 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:34.347140 ip-10-0-142-14 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:03:34.347140 ip-10-0-142-14 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:03:34.349737 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.349652    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:03:34.354568 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354554    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:34.354568 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354568    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354572    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354576    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354579    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354582    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354584    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354586    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354589    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354592    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354594    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354597    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354602    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354606    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354609    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354612    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354615    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354618    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354621    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354624    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:34.354638 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354628    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354632    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354634    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354637    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354640    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354643    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354646    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354649    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354652    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354655    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354658    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354660    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354663    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354665    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354668    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354670    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354672    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354675    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354677    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354680    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:34.355079 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354682    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354685    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354687    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354690    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354692    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354695    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354697    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354700    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354702    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354705    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354707    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354709    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354713    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354717    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354721    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354724    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354727    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354729    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354732    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:34.355571 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354734    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354737    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354739    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354742    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354744    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354746    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354749    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354751    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354754    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354757    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354759    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354762    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354765    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354767    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354770    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354772    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354775    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354777    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354779    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:34.356022 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354782    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354784    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354787    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354789    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354791    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354794    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354797    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354799    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354803    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354806    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354808    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354811    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354814    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354817    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354819    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354822    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354825    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354827    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354830    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354832    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.354835    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:34.356487 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355543    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355551    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355554    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355557    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355560    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355564    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355566    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355569    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355571    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355574    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355577    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355579    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355582    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355585    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355587    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355590    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355592    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355595    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355597    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355601    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:34.357086 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355604    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355606    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355608    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355611    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355613    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355615    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355618    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355620    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355623    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355625    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355627    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355630    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355632    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355634    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355636    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355639    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355641    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355644    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355646    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355649    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:34.357639 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355652    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355654    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355657    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355659    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355661    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355664    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355668    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355671    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355674    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355677    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355680    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355684    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355687    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355691    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355695    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355699    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355701    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355704    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355707    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:34.358139 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355710    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355712    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355716    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355719    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355721    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355723    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355726    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355728    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355731    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355733    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355736    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355738    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355742    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355744    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355747    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355749    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355752    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355754    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355757    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355759    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:34.358613 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355761    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355764    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355766    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355769    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355773    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355776    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355779    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355781    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355783    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355786    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355788    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355791    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355793    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355796    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355799    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355802    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355804    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355808    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355811    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:34.359116 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.355814    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356203    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356210    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356213    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356215    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356219    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356221    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356224    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356227    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356229    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356232    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356234    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356237    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356239    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356242    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356245    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356248    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356251    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356253    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356256    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:34.359584 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356258    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356261    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356263    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356266    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356268    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356271    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356273    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356276    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356278    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356280    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356283    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356285    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356288    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356290    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356292    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356295    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356297    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356300    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356303    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356305    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:34.360058 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356308    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356310    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356312    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356315    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356317    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356319    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356322    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356325    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356327    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356330    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356333    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356335    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356337    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356340    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356342    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356345    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356347    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356350    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356352    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356354    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:34.360550 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356357    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356359    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356361    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356364    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356366    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356369    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356371    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356374    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356376    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356378    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356381    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356384    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356386    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356390    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356392    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356394    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356397    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356399    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356401    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356404    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:34.361021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356411    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356415    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356418    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356420    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356423    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356425    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356429    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356433    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356436    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356441    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356444    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356446    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356449    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356452    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356454    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356457    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356459    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356462    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356464    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:34.361512 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.356466    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356528    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356539    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356544    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356549    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356556    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356561    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356567    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356574    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356577    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356580    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356583    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356587    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356591    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356594    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356597    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356599    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356602    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356605    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356610    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356612    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356615    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356618    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356621    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:03:34.361959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356625    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356628    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356630    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356634    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356637    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356639    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356642    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356645    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356648    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356651    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356654    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356657    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356660    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356662    2562 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356666    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356670    2562 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356673    2562 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356676    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356679    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356683    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356687    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356691    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356694    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356697    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356700    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:03:34.362570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356703    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356706    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356708    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356711    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356714    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356716    2562 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356720    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356723    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356726    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356729    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356732    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356735    2562 flags.go:64] FLAG: --help="false"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356738    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356740    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356743    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356746    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356749    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356752    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356755    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356757    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356760    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356763    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356766    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356769    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:03:34.363211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356771    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356774    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356778    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356781    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356785    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356788    2562 flags.go:64] FLAG: --lock-file=""
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356791    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356793    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356796    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356804    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356807    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356809    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356812    2562 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356815    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356818    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356821    2562 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356823    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356828    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356831    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356835    2562 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356837    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356840    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356843    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356846    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356849    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:03:34.363797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356851    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356854    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356860    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356863    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356866    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356869    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356872    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356877    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356880    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356883    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356888    2562 flags.go:64] FLAG: --port="10250"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356892    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356895    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e678ac67ec3f622"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356898    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356901    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356904    2562 flags.go:64] FLAG: --register-node="true"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356907    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356910    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356913    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356915    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356919    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356922    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356925    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356928    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356931    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:03:34.364400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356934    2562 flags.go:64] FLAG: --runonce="false"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356937    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356940    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356943    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356946    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356948    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356951    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356954    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356957    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356960    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356963    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356965    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356968    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356971    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356974    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356977    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356982    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356986    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356990    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356994    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356996    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.356999    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357001    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357004    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357007    2562 flags.go:64] FLAG: --v="2"
Jul 09 18:03:34.365013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357011    2562 flags.go:64] FLAG: --version="false"
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357015    2562 flags.go:64] FLAG: --vmodule=""
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357018    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.357021    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357151    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357155    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357158    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357161    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357164    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357167    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357171    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357174    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357177    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357179    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357182    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357184    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357187    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357189    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357192    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357194    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:34.365622 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357197    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357199    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357202    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357205    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357208    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357212    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357215    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357218    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357220    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357222    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357225    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357227    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357230    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357232    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357235    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357237    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357240    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357242    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357245    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:34.366158 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357247    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357249    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357252    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357255    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357259    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357262    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357264    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357267    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357269    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357271    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357274    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357276    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357279    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357281    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357283    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357286    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357289    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357293    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357297    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357300    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:34.366616 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357302    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357304    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357307    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357310    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357312    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357314    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357317    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357319    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357322    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357324    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357327    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357329    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357332    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357334    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357336    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357339    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357341    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357344    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357347    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357349    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357351    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:34.367115 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357354    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357357    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357361    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357364    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357367    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357370    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357372    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357375    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357380    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357384    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357386    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357389    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357391    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357393    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357396    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357399    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357401    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357403    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357407    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:34.367607 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357409    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357411    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357414    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.357416    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.358945    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366027    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366044    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366063    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366068    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366138    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366145    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366149    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366152    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366155    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366158    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:34.368049 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366162    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366165    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366168    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366172    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366176    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366179    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366183    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366185    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366188    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366191    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366193    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366196    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366198    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366200    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366203    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366206    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366208    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366211    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366214    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:34.368436 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366216    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366219    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366222    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366224    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366227    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366229    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366232    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366234    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366236    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366239    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366242    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366244    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366246    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366249    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366251    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366254    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366256    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366259    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366261    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:34.368894 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366264    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366268    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366270    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366273    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366276    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366278    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366280    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366283    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366285    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366288    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366290    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366293    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366295    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366298    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366301    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366303    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366306    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366308    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366311    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366314    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:34.369368 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366316    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366318    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366321    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366323    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366326    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366328    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366330    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366333    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366335    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366337    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366340    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366342    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366345    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366348    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366351    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366353    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366356    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366358    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366361    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366364    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:34.369845 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366366    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366369    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366371    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366375    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366378    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366381    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366384    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366387    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366389    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366392    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366394    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366397    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366399    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366401    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:34.370329 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366404    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366409    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366535    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366541    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366545    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366548    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366551    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366554    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366556    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366559    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366561    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366564    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366567    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366569    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:03:34.370662 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366571    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366574    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366576    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366579    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366581    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366583    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366587    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366590    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366592    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366595    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366598    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366601    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366603    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366605    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366608    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366610    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366612    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366615    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366617    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:03:34.370999 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366620    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366623    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366625    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366628    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366631    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366633    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366636    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366638    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366641    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366643    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366646    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366648    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366650    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366653    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366656    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366658    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366660    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366663    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366665    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:03:34.371523 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366667    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366670    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366672    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366675    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366677    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366679    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366682    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366684    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366687    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366689    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366692    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366694    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366696    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366699    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366701    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366703    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366706    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366708    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366710    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366713    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:03:34.371970 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366715    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366717    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366720    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366723    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366725    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366727    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366730    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366732    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366735    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366737    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366740    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366742    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366744    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366747    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366749    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366751    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366754    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366756    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366760    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366763    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:03:34.372447 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366766    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366768    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366770    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366773    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366775    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366778    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366780    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366783    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:34.366785    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366789    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.366882    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.369196    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.370698    2562 server.go:1019] "Starting client certificate rotation"
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.370801    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:03:34.372912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.371654    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:03:34.373404 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.373389    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:03:34.377933 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.377911    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:03:34.390050 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.390030    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:03:34.392247 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.392233    2562 log.go:25] "Validated CRI v1 image API"
Jul 09 18:03:34.393132 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.393117    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:03:34.398028 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.398009    2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8c6f763c-c021-485b-bb49-d8ba95792ff4:/dev/nvme0n1p3 a4df620c-6ed8-43d1-b7fe-eee59c73b690:/dev/nvme0n1p4]
Jul 09 18:03:34.398116 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.398028    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:03:34.403086 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.403070    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:34.404049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.403950    2562 manager.go:216] Machine: {Timestamp:2026-07-09 18:03:34.402468189 +0000 UTC m=+0.373196078 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096121 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd87bb0ed07ba69aace27dd014cfe SystemUUID:ec2cd87b-b0ed-07ba-69aa-ce27dd014cfe BootID:7ffa555b-258f-4677-825c-4e1dcb247c87 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:0b:2f:82:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:0b:2f:82:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:c4:c8:0e:4a:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:03:34.404049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.404047    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:03:34.404157 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.404140    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:03:34.407888 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.407862    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:03:34.408013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.407891    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:03:34.408720 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.408710    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:03:34.408765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.408723    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:03:34.408765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.408735    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:03:34.409417 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.409407    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:03:34.410589 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.410580    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:03:34.415198 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415186    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:03:34.415310 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415300    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:03:34.415343 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415333    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:03:34.415373 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415356    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:03:34.415405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415374    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:03:34.415405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415386    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:03:34.415405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415397    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:03:34.415691 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415681    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:03:34.415737 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.415698    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:03:34.418000 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.417983    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:03:34.420504 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.420491    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:03:34.420548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.420521    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:03:34.421322 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421312    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:03:34.421353 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421328    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:03:34.421353 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421335    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:03:34.421353 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421343    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421355    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421361    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421369    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421374    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421380    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421385    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421393    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:03:34.421429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.421398    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:03:34.423027 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.423018    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:03:34.423071 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.423028    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:03:34.426917 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.426900    2562 server.go:1268] "Started kubelet"
Jul 09 18:03:34.427044 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.426997    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:03:34.427132 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.427041    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:03:34.427183 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.427134    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:03:34.427687 ip-10-0-142-14 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:03:34.429116 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.429082    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:03:34.429353 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.429320    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:03:34.429975 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.429954    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:03:34.430154 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.430130    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:03:34.430241 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.430183    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:03:34.433958 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.433938    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:03:34.437891 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.437871    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:03:34.437891 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.437881    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:34.438035 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.437981    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.438606 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438589    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:03:34.438606 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438589    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:03:34.438723 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438616    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:03:34.438723 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438635    2562 factory.go:55] Registering systemd factory
Jul 09 18:03:34.438723 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438650    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:03:34.438837 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438738    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:03:34.438837 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438745    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:03:34.438910 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438899    2562 factory.go:153] Registering CRI-O factory
Jul 09 18:03:34.438910 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438909    2562 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:03:34.439008 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.438994    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:03:34.439127 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.439051    2562 factory.go:103] Registering Raw factory
Jul 09 18:03:34.439127 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.439075    2562 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:03:34.439997 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.439982    2562 manager.go:349] Starting recovery of all containers
Jul 09 18:03:34.440337 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.439199    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-14.ec2.internal.18c0b1228f1e9bdb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-14.ec2.internal,UID:ip-10-0-142-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-14.ec2.internal,},FirstTimestamp:2026-07-09 18:03:34.426868699 +0000 UTC m=+0.397596589,LastTimestamp:2026-07-09 18:03:34.426868699 +0000 UTC m=+0.397596589,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-14.ec2.internal,}"
Jul 09 18:03:34.450591 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.450573    2562 manager.go:354] Recovery completed
Jul 09 18:03:34.451233 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.451208    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:03:34.451354 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.451321    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:03:34.455615 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.455602    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.457748 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.457733    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.457823 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.457760    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.457823 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.457769    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.458302 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.458289    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:03:34.458352 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.458303    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:03:34.458352 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.458321    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:03:34.458352 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.458296    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.459048 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.459034    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.459129 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.459065    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.459129 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.459075    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.460029 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.459966    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-14.ec2.internal.18c0b12290f5c7d5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-14.ec2.internal,UID:ip-10-0-142-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-14.ec2.internal,},FirstTimestamp:2026-07-09 18:03:34.457747413 +0000 UTC m=+0.428475305,LastTimestamp:2026-07-09 18:03:34.457747413 +0000 UTC m=+0.428475305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-14.ec2.internal,}"
Jul 09 18:03:34.460509 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.460499    2562 policy_none.go:49] "None policy: Start"
Jul 09 18:03:34.460540 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.460514    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:03:34.460540 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.460523    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:03:34.462460 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.462450    2562 policy_none.go:47] "Start"
Jul 09 18:03:34.469284 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.469221    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-14.ec2.internal.18c0b12290f60a23  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-14.ec2.internal,UID:ip-10-0-142-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-14.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-14.ec2.internal,},FirstTimestamp:2026-07-09 18:03:34.457764387 +0000 UTC m=+0.428492278,LastTimestamp:2026-07-09 18:03:34.457764387 +0000 UTC m=+0.428492278,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-14.ec2.internal,}"
Jul 09 18:03:34.479796 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.479741    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-14.ec2.internal.18c0b12290f62cad  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-14.ec2.internal,UID:ip-10-0-142-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-14.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-14.ec2.internal,},FirstTimestamp:2026-07-09 18:03:34.457773229 +0000 UTC m=+0.428501118,LastTimestamp:2026-07-09 18:03:34.457773229 +0000 UTC m=+0.428501118,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-14.ec2.internal,}"
Jul 09 18:03:34.490984 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.490966    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwljj"
Jul 09 18:03:34.493531 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.493475    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-14.ec2.internal.18c0b12290f5c7d5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-14.ec2.internal.18c0b12290f5c7d5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-14.ec2.internal,UID:ip-10-0-142-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-14.ec2.internal,},FirstTimestamp:2026-07-09 18:03:34.457747413 +0000 UTC m=+0.428475305,LastTimestamp:2026-07-09 18:03:34.459046033 +0000 UTC m=+0.429773923,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-14.ec2.internal,}"
Jul 09 18:03:34.496715 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.496701    2562 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:03:34.496774 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.496718    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwljj"
Jul 09 18:03:34.496774 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.496729    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:03:34.496774 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.496739    2562 server.go:87] "Starting device plugin registration server"
Jul 09 18:03:34.496968 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.496954    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:03:34.497055 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.496967    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:03:34.497055 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.497042    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:03:34.497181 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.497125    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:03:34.497181 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.497134    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:03:34.497681 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.497662    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:03:34.497742 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.497715    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.556187 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.556148    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:03:34.558361 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.557413    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:03:34.558361 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.557454    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:03:34.558361 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.557478    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:03:34.558361 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.557515    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:03:34.560062 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.560045    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:34.598073 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.598025    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.599110 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.599078    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.599184 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.599125    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.599184 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.599137    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.599184 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.599158    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.610876 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.610862    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.610954 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.610880    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-14.ec2.internal\": node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.636875 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.636854    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.657943 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.657923    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"]
Jul 09 18:03:34.657995 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.657977    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.659316 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.659295    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.659471 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.659323    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.659471 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.659333    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.661567 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.661555    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.661678 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.661665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.661735 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.661724    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.662224 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.662209    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.662298 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.662234    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.662298 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.662242    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.662298 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.662254    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.662298 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.662264    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.662416 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.662255    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.664451 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.664437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.664522 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.664460    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:34.665179 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.665084    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:34.665179 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.665126    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:34.665179 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.665143    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:34.684600 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.684581    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-14.ec2.internal\" not found" node="ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.688405 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.688391    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-14.ec2.internal\" not found" node="ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.737914 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.737896    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.740161 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.740140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653ef1fe7255847be8d39395467c18fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal\" (UID: \"653ef1fe7255847be8d39395467c18fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.740209 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.740172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e946119a7cd2851de36f572276dc8964-config\") pod \"kube-apiserver-proxy-ip-10-0-142-14.ec2.internal\" (UID: \"e946119a7cd2851de36f572276dc8964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.740209 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.740188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/653ef1fe7255847be8d39395467c18fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal\" (UID: \"653ef1fe7255847be8d39395467c18fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.838819 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.838798    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.840998 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.840984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/653ef1fe7255847be8d39395467c18fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal\" (UID: \"653ef1fe7255847be8d39395467c18fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.841045 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.841006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653ef1fe7255847be8d39395467c18fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal\" (UID: \"653ef1fe7255847be8d39395467c18fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.841045 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.841022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e946119a7cd2851de36f572276dc8964-config\") pod \"kube-apiserver-proxy-ip-10-0-142-14.ec2.internal\" (UID: \"e946119a7cd2851de36f572276dc8964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.841117 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.841064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e946119a7cd2851de36f572276dc8964-config\") pod \"kube-apiserver-proxy-ip-10-0-142-14.ec2.internal\" (UID: \"e946119a7cd2851de36f572276dc8964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.841117 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.841085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/653ef1fe7255847be8d39395467c18fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal\" (UID: \"653ef1fe7255847be8d39395467c18fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.841182 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.841088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653ef1fe7255847be8d39395467c18fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal\" (UID: \"653ef1fe7255847be8d39395467c18fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.939532 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:34.939471    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:34.986810 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.986786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:34.991270 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:34.991254    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:35.040360 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.040330    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.140834 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.140809    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.241444 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.241379    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.341890 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.341862    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.370155 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.370139    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:03:35.370651 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.370275    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:03:35.442307 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.442281    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.489488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.489470    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:03:35.490407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.490379    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:03:35.490528 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.490417    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:03:35.490528 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.490433    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:03:35.504220 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.504159    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:58:35 +0000 UTC" deadline="2027-12-07 19:12:16.19711015 +0000 UTC"
Jul 09 18:03:35.504220 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.504183    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12385h8m40.692931512s"
Jul 09 18:03:35.543372 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.543347    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.643590 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.643555    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.744027 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.743999    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.768052 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.767992    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:35.845109 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:35.845051    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-14.ec2.internal\" not found"
Jul 09 18:03:35.896133 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.896085    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:35.926220 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.926198    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:35.936746 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:35.936721    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode946119a7cd2851de36f572276dc8964.slice/crio-003c78c389faf62a83f9e0689acbf3a2e91e8ba93efd0a97236af427c4756f25 WatchSource:0}: Error finding container 003c78c389faf62a83f9e0689acbf3a2e91e8ba93efd0a97236af427c4756f25: Status 404 returned error can't find the container with id 003c78c389faf62a83f9e0689acbf3a2e91e8ba93efd0a97236af427c4756f25
Jul 09 18:03:35.938887 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.938871    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:35.943837 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.943820    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:03:35.949632 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:35.949612    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653ef1fe7255847be8d39395467c18fa.slice/crio-c320a039028ea37dda9a33ba30c7dd1756b713a641811c4543e9cec5c353f533 WatchSource:0}: Error finding container c320a039028ea37dda9a33ba30c7dd1756b713a641811c4543e9cec5c353f533: Status 404 returned error can't find the container with id c320a039028ea37dda9a33ba30c7dd1756b713a641811c4543e9cec5c353f533
Jul 09 18:03:35.951584 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.951571    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:03:35.952511 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.952493    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal"
Jul 09 18:03:35.960549 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:35.960534    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:03:36.238363 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.238279    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:03:36.418897 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.418850    2562 apiserver.go:52] "Watching apiserver"
Jul 09 18:03:36.426833 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.426813    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:03:36.428724 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.428702    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-jvsm2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal","openshift-multus/network-metrics-daemon-846sw","openshift-ovn-kubernetes/ovnkube-node-jrngm","kube-system/konnectivity-agent-96g7t","kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs","openshift-cluster-node-tuning-operator/tuned-rfwpj","openshift-multus/multus-additional-cni-plugins-7xqwq","openshift-multus/multus-pmgzp","openshift-network-diagnostics/network-check-target-vbhk9","openshift-network-operator/iptables-alerter-ztl92","kube-system/global-pull-secret-syncer-mb25c","openshift-dns/node-resolver-j45n7"]
Jul 09 18:03:36.431633 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.431615    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.433901 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.433878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.434006 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.433967    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:36.435207 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.435170    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.435316 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.435273    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.435473 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.435454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:03:36.435545 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.435464    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-npbbg\""
Jul 09 18:03:36.436176 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.436146    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:36.436273 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.436242    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:36.438294 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.438252    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:03:36.438579 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.438558    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.440938 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.440921    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.441403 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.441383    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:03:36.441502 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.441487    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xb46k\""
Jul 09 18:03:36.441907 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.441892    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.441907 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.441902    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:03:36.442013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.442003    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:03:36.442551 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.442535    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:03:36.442637 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.442601    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.443517 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.443500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.443888 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.443867    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:03:36.443988 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.443874    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:03:36.444157 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.444144    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r487q\""
Jul 09 18:03:36.446192 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.446160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.446529 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.446509    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.446932    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.447261    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-429pr\""
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.449581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-ovnkube-script-lib\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.449722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.449903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3502d811-6ad3-4ac5-a954-b170184e50e4-ovn-node-metrics-cert\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.449984    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.449988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-dbus\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mb6\" (UniqueName: \"kubernetes.io/projected/80b70a46-af4d-46ce-9ef4-443bac043d9d-kube-api-access-n8mb6\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450142    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:03:36.450173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-systemd-units\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450198    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-log-socket\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450241    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7ng8c\""
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-ovn\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-cni-bin\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-etc-selinux\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-sys-fs\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-kubelet\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-etc-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450614    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-cni-netd\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-ovnkube-config\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-env-overrides\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.450847 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-run-netns\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-var-lib-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5-konnectivity-ca\") pod \"konnectivity-agent-96g7t\" (UID: \"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5\") " pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.450996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-socket-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-device-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-kubelet-config\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngwb\" (UniqueName: \"kubernetes.io/projected/3502d811-6ad3-4ac5-a954-b170184e50e4-kube-api-access-6ngwb\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-node-log\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-registration-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48px\" (UniqueName: \"kubernetes.io/projected/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-kube-api-access-g48px\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-slash\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-systemd\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.451429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.451245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5-agent-certs\") pod \"konnectivity-agent-96g7t\" (UID: \"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5\") " pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.452623 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.452601    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.454078 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.454061    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:03:36.455770 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.455444    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.455770 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.455462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.455770 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.455491    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.455770 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.455521    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jfc54\""
Jul 09 18:03:36.455770 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.455470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:03:36.457906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.457886    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.458326 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.458305    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4dmcn\""
Jul 09 18:03:36.458431 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.458411    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.458600 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.458306    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.460459 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.460441    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:36.460559 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.460472    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:03:36.460559 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.460494    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:36.460999 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.460981    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-t8krb\""
Jul 09 18:03:36.463140 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.463122    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.468059 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.468038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:03:36.468174 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.468137    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:03:36.468174 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.468153    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:03:36.468304 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.468159    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mptk2\""
Jul 09 18:03:36.482625 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.482610    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wcfk9"
Jul 09 18:03:36.494554 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.494539    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wcfk9"
Jul 09 18:03:36.539906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.539888    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:03:36.552133 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.552227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-systemd-units\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a66269-f70d-45a4-9bf4-0599903b8af8-tmp-dir\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.552227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-host\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.552432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-systemd-units\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-conf-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.552432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-ovnkube-config\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5zx\" (UniqueName: \"kubernetes.io/projected/13cb5c00-60d1-4a33-8302-cac1198f9c1f-kube-api-access-bc5zx\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.552432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-lib-modules\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-etc-selinux\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-multus-certs\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-etc-selinux\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-cni-netd\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da0f105-22fb-487f-8bd9-37922c1d5ac5-tmp\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-etc-kubernetes\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.552732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-cni-netd\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5-konnectivity-ca\") pod \"konnectivity-agent-96g7t\" (UID: \"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5\") " pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-var-lib-kubelet\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-hostroot\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-device-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngwb\" (UniqueName: \"kubernetes.io/projected/3502d811-6ad3-4ac5-a954-b170184e50e4-kube-api-access-6ngwb\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzd2\" (UniqueName: \"kubernetes.io/projected/11a66269-f70d-45a4-9bf4-0599903b8af8-kube-api-access-6gzd2\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-device-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.552963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vffv5\" (UniqueName: \"kubernetes.io/projected/6abe54c6-a745-48bf-9e63-660221669af7-kube-api-access-vffv5\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-systemd\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/061e0238-d810-4382-82cc-74c9a103a439-iptables-alerter-script\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdgb\" (UniqueName: \"kubernetes.io/projected/061e0238-d810-4382-82cc-74c9a103a439-kube-api-access-kxdgb\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysctl-conf\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-systemd\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.553407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5-konnectivity-ca\") pod \"konnectivity-agent-96g7t\" (UID: \"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5\") " pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-cni-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-os-release\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-cnibin\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshpt\" (UniqueName: \"kubernetes.io/projected/ec79ba9f-c9c8-4c31-911f-85e59598901f-kube-api-access-vshpt\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-kubernetes\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-dbus\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mb6\" (UniqueName: \"kubernetes.io/projected/80b70a46-af4d-46ce-9ef4-443bac043d9d-kube-api-access-n8mb6\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-log-socket\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13cb5c00-60d1-4a33-8302-cac1198f9c1f-host\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-env-overrides\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13cb5c00-60d1-4a33-8302-cac1198f9c1f-serviceca\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-modprobe-d\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-dbus\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-log-socket\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-ovn\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-run-ovn\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-cni-bin\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-sys-fs\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-kubelet\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-etc-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-cni-bin\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11a66269-f70d-45a4-9bf4-0599903b8af8-hosts-file\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysctl-d\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-kubelet\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-sys-fs\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-etc-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-tuned\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-cnibin\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6abe54c6-a745-48bf-9e63-660221669af7-cni-binary-copy\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-socket-dir-parent\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-k8s-cni-cncf-io\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.553633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-ovnkube-config\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.554990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-run-netns\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-var-lib-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-sys\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-run-netns\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-var-lib-openvswitch\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6abe54c6-a745-48bf-9e63-660221669af7-multus-daemon-config\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-system-cni-dir\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.554481    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-socket-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.554561    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:37.054524628 +0000 UTC m=+3.025252533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-env-overrides\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-kubelet-config\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.555784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-socket-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysconfig\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-systemd\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-kubelet-config\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-run\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlbp\" (UniqueName: \"kubernetes.io/projected/9da0f105-22fb-487f-8bd9-37922c1d5ac5-kube-api-access-kqlbp\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-system-cni-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-kubelet\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-node-log\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.554865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-node-log\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-registration-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-registration-dir\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48px\" (UniqueName: \"kubernetes.io/projected/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-kube-api-access-g48px\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-slash\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5-agent-certs\") pod \"konnectivity-agent-96g7t\" (UID: \"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5\") " pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/061e0238-d810-4382-82cc-74c9a103a439-host-slash\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555894    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:03:36.556538 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-ovnkube-script-lib\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.555895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3502d811-6ad3-4ac5-a954-b170184e50e4-host-slash\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3502d811-6ad3-4ac5-a954-b170184e50e4-ovn-node-metrics-cert\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-netns\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-cni-bin\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-cni-multus\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-os-release\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.556369    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.556422    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:37.056398615 +0000 UTC m=+3.027126508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:36.557346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.556652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3502d811-6ad3-4ac5-a954-b170184e50e4-ovnkube-script-lib\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.560505 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.560466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3502d811-6ad3-4ac5-a954-b170184e50e4-ovn-node-metrics-cert\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.560602 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.560561    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5-agent-certs\") pod \"konnectivity-agent-96g7t\" (UID: \"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5\") " pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.562948 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.562906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal" event={"ID":"653ef1fe7255847be8d39395467c18fa","Type":"ContainerStarted","Data":"c320a039028ea37dda9a33ba30c7dd1756b713a641811c4543e9cec5c353f533"}
Jul 09 18:03:36.563387 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.563333    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngwb\" (UniqueName: \"kubernetes.io/projected/3502d811-6ad3-4ac5-a954-b170184e50e4-kube-api-access-6ngwb\") pod \"ovnkube-node-jrngm\" (UID: \"3502d811-6ad3-4ac5-a954-b170184e50e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.563790 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.563757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mb6\" (UniqueName: \"kubernetes.io/projected/80b70a46-af4d-46ce-9ef4-443bac043d9d-kube-api-access-n8mb6\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:36.564219 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.564193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal" event={"ID":"e946119a7cd2851de36f572276dc8964","Type":"ContainerStarted","Data":"003c78c389faf62a83f9e0689acbf3a2e91e8ba93efd0a97236af427c4756f25"}
Jul 09 18:03:36.564647 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.564630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48px\" (UniqueName: \"kubernetes.io/projected/7a3c4b01-1484-4ddf-9f18-5c0d0b24462a-kube-api-access-g48px\") pod \"aws-ebs-csi-driver-node-dqsbs\" (UID: \"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.657361 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/061e0238-d810-4382-82cc-74c9a103a439-iptables-alerter-script\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.657488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdgb\" (UniqueName: \"kubernetes.io/projected/061e0238-d810-4382-82cc-74c9a103a439-kube-api-access-kxdgb\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.657488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysctl-conf\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.657488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.657488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-cni-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.657488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-os-release\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-cnibin\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshpt\" (UniqueName: \"kubernetes.io/projected/ec79ba9f-c9c8-4c31-911f-85e59598901f-kube-api-access-vshpt\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-kubernetes\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13cb5c00-60d1-4a33-8302-cac1198f9c1f-host\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13cb5c00-60d1-4a33-8302-cac1198f9c1f-serviceca\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-modprobe-d\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11a66269-f70d-45a4-9bf4-0599903b8af8-hosts-file\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysctl-d\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-tuned\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-cnibin\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6abe54c6-a745-48bf-9e63-660221669af7-cni-binary-copy\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-socket-dir-parent\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-k8s-cni-cncf-io\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.657772 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-sys\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6abe54c6-a745-48bf-9e63-660221669af7-multus-daemon-config\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-system-cni-dir\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657856    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-system-cni-dir\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/061e0238-d810-4382-82cc-74c9a103a439-iptables-alerter-script\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-kubernetes\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-cnibin\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.657996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11a66269-f70d-45a4-9bf4-0599903b8af8-hosts-file\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-modprobe-d\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysctl-conf\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-os-release\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-cni-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-sys\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysctl-d\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-socket-dir-parent\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-cnibin\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13cb5c00-60d1-4a33-8302-cac1198f9c1f-serviceca\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-k8s-cni-cncf-io\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.658548 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13cb5c00-60d1-4a33-8302-cac1198f9c1f-host\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6abe54c6-a745-48bf-9e63-660221669af7-cni-binary-copy\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysconfig\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-systemd\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-run\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlbp\" (UniqueName: \"kubernetes.io/projected/9da0f105-22fb-487f-8bd9-37922c1d5ac5-kube-api-access-kqlbp\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-systemd\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-system-cni-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-sysconfig\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-system-cni-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-run\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-kubelet\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6abe54c6-a745-48bf-9e63-660221669af7-multus-daemon-config\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/061e0238-d810-4382-82cc-74c9a103a439-host-slash\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/061e0238-d810-4382-82cc-74c9a103a439-host-slash\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658882    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-netns\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.659501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-kubelet\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-cni-bin\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-cni-bin\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-cni-multus\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-netns\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-os-release\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.658986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-var-lib-cni-multus\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a66269-f70d-45a4-9bf4-0599903b8af8-tmp-dir\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-os-release\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-host\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659113    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-host\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-conf-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5zx\" (UniqueName: \"kubernetes.io/projected/13cb5c00-60d1-4a33-8302-cac1198f9c1f-kube-api-access-bc5zx\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-lib-modules\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-multus-certs\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da0f105-22fb-487f-8bd9-37922c1d5ac5-tmp\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.660581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-etc-kubernetes\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-var-lib-kubelet\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11a66269-f70d-45a4-9bf4-0599903b8af8-tmp-dir\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-hostroot\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-hostroot\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzd2\" (UniqueName: \"kubernetes.io/projected/11a66269-f70d-45a4-9bf4-0599903b8af8-kube-api-access-6gzd2\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vffv5\" (UniqueName: \"kubernetes.io/projected/6abe54c6-a745-48bf-9e63-660221669af7-kube-api-access-vffv5\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-lib-modules\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-multus-conf-dir\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659561    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec79ba9f-c9c8-4c31-911f-85e59598901f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659575    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-etc-kubernetes\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec79ba9f-c9c8-4c31-911f-85e59598901f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6abe54c6-a745-48bf-9e63-660221669af7-host-run-multus-certs\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9da0f105-22fb-487f-8bd9-37922c1d5ac5-var-lib-kubelet\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.659493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:36.661592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.660847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9da0f105-22fb-487f-8bd9-37922c1d5ac5-etc-tuned\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.662349 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.662081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9da0f105-22fb-487f-8bd9-37922c1d5ac5-tmp\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.669047 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.669022    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:36.669047 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.669049    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:36.669229 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.669063    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:36.669229 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:36.669129    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:03:37.169112068 +0000 UTC m=+3.139839972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:36.672486 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.672269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5zx\" (UniqueName: \"kubernetes.io/projected/13cb5c00-60d1-4a33-8302-cac1198f9c1f-kube-api-access-bc5zx\") pod \"node-ca-jvsm2\" (UID: \"13cb5c00-60d1-4a33-8302-cac1198f9c1f\") " pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.672486 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.672448    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzd2\" (UniqueName: \"kubernetes.io/projected/11a66269-f70d-45a4-9bf4-0599903b8af8-kube-api-access-6gzd2\") pod \"node-resolver-j45n7\" (UID: \"11a66269-f70d-45a4-9bf4-0599903b8af8\") " pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.672641 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.672509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshpt\" (UniqueName: \"kubernetes.io/projected/ec79ba9f-c9c8-4c31-911f-85e59598901f-kube-api-access-vshpt\") pod \"multus-additional-cni-plugins-7xqwq\" (UID: \"ec79ba9f-c9c8-4c31-911f-85e59598901f\") " pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.672819 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.672795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdgb\" (UniqueName: \"kubernetes.io/projected/061e0238-d810-4382-82cc-74c9a103a439-kube-api-access-kxdgb\") pod \"iptables-alerter-ztl92\" (UID: \"061e0238-d810-4382-82cc-74c9a103a439\") " pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.673400 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.673383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlbp\" (UniqueName: \"kubernetes.io/projected/9da0f105-22fb-487f-8bd9-37922c1d5ac5-kube-api-access-kqlbp\") pod \"tuned-rfwpj\" (UID: \"9da0f105-22fb-487f-8bd9-37922c1d5ac5\") " pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.674344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.674320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vffv5\" (UniqueName: \"kubernetes.io/projected/6abe54c6-a745-48bf-9e63-660221669af7-kube-api-access-vffv5\") pod \"multus-pmgzp\" (UID: \"6abe54c6-a745-48bf-9e63-660221669af7\") " pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.740609 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.740581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs"
Jul 09 18:03:36.754187 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.754145    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3c4b01_1484_4ddf_9f18_5c0d0b24462a.slice/crio-06d59f2273ca38ed500437584b51201faaf0c840e5775edb99394087b0bc1b1a WatchSource:0}: Error finding container 06d59f2273ca38ed500437584b51201faaf0c840e5775edb99394087b0bc1b1a: Status 404 returned error can't find the container with id 06d59f2273ca38ed500437584b51201faaf0c840e5775edb99394087b0bc1b1a
Jul 09 18:03:36.754299 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.754280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:03:36.759546 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.759526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:36.764317 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.764288    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3502d811_6ad3_4ac5_a954_b170184e50e4.slice/crio-1002a2df3f0b6e2d3beb1428e8586dd39ca2ad302d23ce72e7496e08c85ec780 WatchSource:0}: Error finding container 1002a2df3f0b6e2d3beb1428e8586dd39ca2ad302d23ce72e7496e08c85ec780: Status 404 returned error can't find the container with id 1002a2df3f0b6e2d3beb1428e8586dd39ca2ad302d23ce72e7496e08c85ec780
Jul 09 18:03:36.765059 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.765036    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j45n7"
Jul 09 18:03:36.771676 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.771646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7xqwq"
Jul 09 18:03:36.773582 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.773559    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aab7f7d_2c9f_4812_b5b7_7b07d7151eb5.slice/crio-0ad8dc4ce3f9069985529dcec936f172c344d134892eca8f4309c18374a5ab43 WatchSource:0}: Error finding container 0ad8dc4ce3f9069985529dcec936f172c344d134892eca8f4309c18374a5ab43: Status 404 returned error can't find the container with id 0ad8dc4ce3f9069985529dcec936f172c344d134892eca8f4309c18374a5ab43
Jul 09 18:03:36.776655 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.776483    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a66269_f70d_45a4_9bf4_0599903b8af8.slice/crio-29f106f1facbeabe0613b04552d777710d08902501093f6a30198759161dc78e WatchSource:0}: Error finding container 29f106f1facbeabe0613b04552d777710d08902501093f6a30198759161dc78e: Status 404 returned error can't find the container with id 29f106f1facbeabe0613b04552d777710d08902501093f6a30198759161dc78e
Jul 09 18:03:36.776655 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.776580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvsm2"
Jul 09 18:03:36.783342 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.783319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rfwpj"
Jul 09 18:03:36.787118 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.787096    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec79ba9f_c9c8_4c31_911f_85e59598901f.slice/crio-da0e4b4a6d40ad15df7d344442fed70e0b4d4059fe75659964af7517f7601d6f WatchSource:0}: Error finding container da0e4b4a6d40ad15df7d344442fed70e0b4d4059fe75659964af7517f7601d6f: Status 404 returned error can't find the container with id da0e4b4a6d40ad15df7d344442fed70e0b4d4059fe75659964af7517f7601d6f
Jul 09 18:03:36.789344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.789322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pmgzp"
Jul 09 18:03:36.794199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:36.794157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ztl92"
Jul 09 18:03:36.798888 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.798852    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da0f105_22fb_487f_8bd9_37922c1d5ac5.slice/crio-f9e3ff61c3694b8420bd779f2280417d7b3b2243ff9822f41d33f962a22cceb6 WatchSource:0}: Error finding container f9e3ff61c3694b8420bd779f2280417d7b3b2243ff9822f41d33f962a22cceb6: Status 404 returned error can't find the container with id f9e3ff61c3694b8420bd779f2280417d7b3b2243ff9822f41d33f962a22cceb6
Jul 09 18:03:36.803727 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.803701    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe54c6_a745_48bf_9e63_660221669af7.slice/crio-653388a536abbe2f9ae78f214b323dc736dbfa5f112173a13ecfe81a95a7b38b WatchSource:0}: Error finding container 653388a536abbe2f9ae78f214b323dc736dbfa5f112173a13ecfe81a95a7b38b: Status 404 returned error can't find the container with id 653388a536abbe2f9ae78f214b323dc736dbfa5f112173a13ecfe81a95a7b38b
Jul 09 18:03:36.814021 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:03:36.813996    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e0238_d810_4382_82cc_74c9a103a439.slice/crio-f1d86bb60d4ea122b3db1bbe8e8cd683ce9c0edc832ef608d06e82e255ed49e3 WatchSource:0}: Error finding container f1d86bb60d4ea122b3db1bbe8e8cd683ce9c0edc832ef608d06e82e255ed49e3: Status 404 returned error can't find the container with id f1d86bb60d4ea122b3db1bbe8e8cd683ce9c0edc832ef608d06e82e255ed49e3
Jul 09 18:03:37.063827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.063728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:37.063827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.063793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:37.064055 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.063916    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:37.064055 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.063917    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:37.064055 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.063977    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:38.063961833 +0000 UTC m=+4.034689730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:37.064055 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.064005    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:38.063984141 +0000 UTC m=+4.034712025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:37.265973 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.265321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:37.265973 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.265500    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:37.265973 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.265518    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:37.265973 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.265532    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:37.265973 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.265596    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:03:38.265576406 +0000 UTC m=+4.236304308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:37.498556 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.498443    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:37 +0000 UTC" deadline="2028-02-14 06:37:21.360219001 +0000 UTC"
Jul 09 18:03:37.498556 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.498509    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14028h33m43.861714385s"
Jul 09 18:03:37.558743 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.558017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:37.558743 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.558171    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:37.558743 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.558604    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:37.558743 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:37.558696    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:37.568044 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.567985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs" event={"ID":"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a","Type":"ContainerStarted","Data":"06d59f2273ca38ed500437584b51201faaf0c840e5775edb99394087b0bc1b1a"}
Jul 09 18:03:37.569449 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.569418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ztl92" event={"ID":"061e0238-d810-4382-82cc-74c9a103a439","Type":"ContainerStarted","Data":"f1d86bb60d4ea122b3db1bbe8e8cd683ce9c0edc832ef608d06e82e255ed49e3"}
Jul 09 18:03:37.570758 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.570707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pmgzp" event={"ID":"6abe54c6-a745-48bf-9e63-660221669af7","Type":"ContainerStarted","Data":"653388a536abbe2f9ae78f214b323dc736dbfa5f112173a13ecfe81a95a7b38b"}
Jul 09 18:03:37.572730 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.572701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rfwpj" event={"ID":"9da0f105-22fb-487f-8bd9-37922c1d5ac5","Type":"ContainerStarted","Data":"f9e3ff61c3694b8420bd779f2280417d7b3b2243ff9822f41d33f962a22cceb6"}
Jul 09 18:03:37.574272 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.574227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96g7t" event={"ID":"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5","Type":"ContainerStarted","Data":"0ad8dc4ce3f9069985529dcec936f172c344d134892eca8f4309c18374a5ab43"}
Jul 09 18:03:37.575862 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.575835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"1002a2df3f0b6e2d3beb1428e8586dd39ca2ad302d23ce72e7496e08c85ec780"}
Jul 09 18:03:37.577866 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.577841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvsm2" event={"ID":"13cb5c00-60d1-4a33-8302-cac1198f9c1f","Type":"ContainerStarted","Data":"a9c349151100579b405e807f0adc6519b3402edfe3d20b12b6268d865f3a2a1d"}
Jul 09 18:03:37.579513 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.579484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerStarted","Data":"da0e4b4a6d40ad15df7d344442fed70e0b4d4059fe75659964af7517f7601d6f"}
Jul 09 18:03:37.580988 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:37.580959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j45n7" event={"ID":"11a66269-f70d-45a4-9bf4-0599903b8af8","Type":"ContainerStarted","Data":"29f106f1facbeabe0613b04552d777710d08902501093f6a30198759161dc78e"}
Jul 09 18:03:38.072795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:38.072737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:38.072795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:38.072802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:38.073025 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.072932    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:38.073025 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.072999    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:40.072977281 +0000 UTC m=+6.043705185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:38.073388 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.073372    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:38.073482 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.073417    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:40.073404089 +0000 UTC m=+6.044131968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:38.274945 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:38.274905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:38.275207 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.275164    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:38.275207 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.275194    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:38.275207 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.275209    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:38.275406 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.275284    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:03:40.275262282 +0000 UTC m=+6.245990163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:38.499410 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:38.499275    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:58:37 +0000 UTC" deadline="2028-03-18 13:50:26.424465516 +0000 UTC"
Jul 09 18:03:38.499410 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:38.499321    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14827h46m47.925149988s"
Jul 09 18:03:38.559082 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:38.558547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:38.559082 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:38.558710    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:39.557751 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:39.557718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:39.558290 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:39.557863    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:39.558447 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:39.558410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:39.558537 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:39.558514    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:40.093384 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:40.093339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:40.093587 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:40.093413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:40.093650 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.093589    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:40.093709 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.093659    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:44.093639062 +0000 UTC m=+10.064366944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:40.094109 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.094085    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:40.094229 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.094147    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:44.0941311 +0000 UTC m=+10.064858982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:40.295456 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:40.295416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:40.295635 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.295594    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:40.295635 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.295619    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:40.295635 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.295633    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:40.295809 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.295695    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:03:44.295676489 +0000 UTC m=+10.266404373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:40.561137 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:40.560339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:40.561137 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:40.560499    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:41.558952 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:41.558383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:41.558952 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:41.558500    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:41.558952 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:41.558566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:41.558952 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:41.558633    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:42.558088 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:42.558052    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:42.558581 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:42.558229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:43.558198 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:43.558165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:43.558663 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:43.558165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:43.558663 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:43.558309    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:43.558663 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:43.558389    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:44.130701 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:44.130662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:44.130853 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:44.130781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:44.130853 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.130836    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:44.130923 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.130900    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:03:52.130880017 +0000 UTC m=+18.101607909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:44.130923 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.130903    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:44.130996 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.130962    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:03:52.130944642 +0000 UTC m=+18.101672534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:44.332452 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:44.332406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:44.332631 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.332607    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:44.332631 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.332634    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:44.332821 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.332648    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:44.332821 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.332715    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:03:52.332695459 +0000 UTC m=+18.303423336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:44.559340 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:44.559264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:44.559785 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:44.559394    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:45.558504 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:45.558475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:45.558696 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:45.558591    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:45.558696 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:45.558656    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:45.558815 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:45.558770    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:46.558551 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:46.558023    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:46.558551 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:46.558187    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:47.558784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:47.558167    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:47.558784 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:47.558288    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:47.558784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:47.558367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:47.558784 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:47.558439    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:48.558414 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:48.558331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:48.558557 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:48.558489    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:48.623985 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:48.623952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal" event={"ID":"e946119a7cd2851de36f572276dc8964","Type":"ContainerStarted","Data":"b8a83158003e8c8d76d5c5264a16faf77035310a8dc79de4398a0f43396cbe4b"}
Jul 09 18:03:48.625867 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:48.625835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rfwpj" event={"ID":"9da0f105-22fb-487f-8bd9-37922c1d5ac5","Type":"ContainerStarted","Data":"f8c9399865bf31a2b4ab4db2d6d192dc5367fad66ce3873707d820dc4dcce2a9"}
Jul 09 18:03:48.641696 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:48.641646    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-14.ec2.internal" podStartSLOduration=12.641630073 podStartE2EDuration="12.641630073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.641412749 +0000 UTC m=+14.612140650" watchObservedRunningTime="2026-07-09 18:03:48.641630073 +0000 UTC m=+14.612357973"
Jul 09 18:03:49.558643 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:49.558415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:49.558779 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:49.558423    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:49.558779 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:49.558684    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:49.558858 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:49.558774    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:50.558019 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.557984    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:50.558542 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:50.558149    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:50.632758 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.632723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal" event={"ID":"653ef1fe7255847be8d39395467c18fa","Type":"ContainerStarted","Data":"8aa7db4831b0658d552387695dc26dc7c0a1731340c783c08378f3934ee0c7f0"}
Jul 09 18:03:50.634266 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.634232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvsm2" event={"ID":"13cb5c00-60d1-4a33-8302-cac1198f9c1f","Type":"ContainerStarted","Data":"120e9c7a243af44a31a32ed9377325888bd31e40a20276f23449fd685f1d936b"}
Jul 09 18:03:50.635827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.635805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerStarted","Data":"e4957ae516d94e375bc474677319d2e291f03cabced2d2cbf9aa7bdadaa90192"}
Jul 09 18:03:50.637531 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.637258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j45n7" event={"ID":"11a66269-f70d-45a4-9bf4-0599903b8af8","Type":"ContainerStarted","Data":"776846b58f3a50642bc845c47081903675e87bd8072c6c39ad723ef30f4676b9"}
Jul 09 18:03:50.639554 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.639533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs" event={"ID":"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a","Type":"ContainerStarted","Data":"3d6005a4dcb1f00d4ed6098bea1aa3692975f97434b2c5702c2023bdaaf96c78"}
Jul 09 18:03:50.641213 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.641192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ztl92" event={"ID":"061e0238-d810-4382-82cc-74c9a103a439","Type":"ContainerStarted","Data":"9b43deac7ea6b441ab289bac5ac87cb0dd55568f7442d334b311e9c06488c9c5"}
Jul 09 18:03:50.642746 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.642725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96g7t" event={"ID":"1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5","Type":"ContainerStarted","Data":"167d711fb53b02bf5af92c1276a969be7cc48c330cf6c9075109fe9015e5ca99"}
Jul 09 18:03:50.650155 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.650119    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rfwpj" podStartSLOduration=4.18521683 podStartE2EDuration="15.650089532s" totalImagesPullingTime="11.464872702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:48.66504539 +0000 UTC m=+14.635773292" watchObservedRunningTime="2026-07-09 18:03:50.650089532 +0000 UTC m=+16.620817432"
Jul 09 18:03:50.666501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.666466    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-96g7t" podStartSLOduration=4.225664788 podStartE2EDuration="15.666454594s" totalImagesPullingTime="11.440789806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:50.666384546 +0000 UTC m=+16.637112435" watchObservedRunningTime="2026-07-09 18:03:50.666454594 +0000 UTC m=+16.637182483"
Jul 09 18:03:50.722105 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.722042    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j45n7" podStartSLOduration=4.233909796 podStartE2EDuration="15.722025205s" totalImagesPullingTime="11.488115409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:50.721582614 +0000 UTC m=+16.692310515" watchObservedRunningTime="2026-07-09 18:03:50.722025205 +0000 UTC m=+16.692753105"
Jul 09 18:03:50.722278 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.722145    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jvsm2" podStartSLOduration=4.236646114 podStartE2EDuration="15.722138386s" totalImagesPullingTime="11.485492272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:50.705779687 +0000 UTC m=+16.676507588" watchObservedRunningTime="2026-07-09 18:03:50.722138386 +0000 UTC m=+16.692866285"
Jul 09 18:03:50.737813 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:50.737765    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ztl92" podStartSLOduration=4.117366804 podStartE2EDuration="15.737750071s" totalImagesPullingTime="11.620383267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:50.737427406 +0000 UTC m=+16.708155305" watchObservedRunningTime="2026-07-09 18:03:50.737750071 +0000 UTC m=+16.708477972"
Jul 09 18:03:51.557920 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:51.557845    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:51.557920 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:51.557879    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:51.558464 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:51.557964    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:51.558464 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:51.558123    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:52.193173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:52.193136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:52.193173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:52.193188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:52.193526 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.193299    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:52.193526 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.193299    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:52.193526 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.193361    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:08.193345653 +0000 UTC m=+34.164073532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:03:52.193526 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.193375    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:08.193369116 +0000 UTC m=+34.164096993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:03:52.394788 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:52.394756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:52.394964 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.394940    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:03:52.394964 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.394959    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:03:52.395069 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.394968    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:52.395069 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.395026    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:08.395009822 +0000 UTC m=+34.365737719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:03:52.560417 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:52.560338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:52.560997 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:52.560475    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:53.558346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:53.558305    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:53.558513 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:53.558439    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:53.558585 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:53.558564    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:53.558693 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:53.558675    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:53.653413 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:53.653376    2562 generic.go:356] "Generic (PLEG): container finished" podID="653ef1fe7255847be8d39395467c18fa" containerID="8aa7db4831b0658d552387695dc26dc7c0a1731340c783c08378f3934ee0c7f0" exitCode=0
Jul 09 18:03:53.653906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:53.653469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal" event={"ID":"653ef1fe7255847be8d39395467c18fa","Type":"ContainerDied","Data":"8aa7db4831b0658d552387695dc26dc7c0a1731340c783c08378f3934ee0c7f0"}
Jul 09 18:03:53.655941 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:53.655916    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec79ba9f-c9c8-4c31-911f-85e59598901f" containerID="e4957ae516d94e375bc474677319d2e291f03cabced2d2cbf9aa7bdadaa90192" exitCode=0
Jul 09 18:03:53.656060 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:53.655958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerDied","Data":"e4957ae516d94e375bc474677319d2e291f03cabced2d2cbf9aa7bdadaa90192"}
Jul 09 18:03:54.118908 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:54.118818    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:54.119669 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:54.119642    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:54.559254 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:54.558983    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:54.559413 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:54.559257    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:55.558628 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:55.558578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:55.559046 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:55.558724    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:55.559114 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:55.558578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:55.559189 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:55.559159    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:56.558186 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:56.558147    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:56.558348 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:56.558306    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:57.558341 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:57.558312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:57.558897 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:57.558312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:57.558897 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:57.558440    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:57.558897 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:57.558512    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:58.081655 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.081110    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:03:58.508206 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.508083    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:03:58.081270486Z","UUID":"1d9b6e8c-6d95-437a-9dde-762ad8d89135","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:03:58.509853 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.509828    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:03:58.509982 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.509872    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:03:58.558227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.558190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:03:58.558368 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:58.558325    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:03:58.669309 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.669274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs" event={"ID":"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a","Type":"ContainerStarted","Data":"7b57af91a8c8615df524d947c638c0b2bb7caa61bbe2757ea093c468d9e51396"}
Jul 09 18:03:58.671289 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.671251    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pmgzp" event={"ID":"6abe54c6-a745-48bf-9e63-660221669af7","Type":"ContainerStarted","Data":"7e98730c7a7db9f18bd285414562e940ef28a091ab357d7c6add41c36347468c"}
Jul 09 18:03:58.674646 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.674622    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:03:58.675032 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675001    2562 generic.go:356] "Generic (PLEG): container finished" podID="3502d811-6ad3-4ac5-a954-b170184e50e4" containerID="4f160486d5f34dde1a1a151a2a9558f8e7b2830cf6dbee7e9587368de6d84cc3" exitCode=1
Jul 09 18:03:58.675186 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"dff9cddc625fc1b7598661e601796cd25a768ff67004389df06e2dae1ed5e9c5"}
Jul 09 18:03:58.675255 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"9989b3fff9300a58ada67366b5a66570c2759512092dbb9b18b69deba8eab00f"}
Jul 09 18:03:58.675255 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"ddd6bf5d927eaa8361e6f96390360f0509eee861ed5da43bbf189a0e88732268"}
Jul 09 18:03:58.675391 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"1c907842d3775ad60f627bb46d67a3a5ae6a2b911a25200a7c4754d67312ae6b"}
Jul 09 18:03:58.675391 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerDied","Data":"4f160486d5f34dde1a1a151a2a9558f8e7b2830cf6dbee7e9587368de6d84cc3"}
Jul 09 18:03:58.675484 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.675398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"95989527b8942accaea4fdba2cb6a2bab57dc72141adf7a9732a4e67651c481c"}
Jul 09 18:03:58.677707 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.677687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal" event={"ID":"653ef1fe7255847be8d39395467c18fa","Type":"ContainerStarted","Data":"28f919334f349d07c79310566aade82f935946fec0aa07355f55b3fb1c9f8a7e"}
Jul 09 18:03:58.690803 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:58.690754    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pmgzp" podStartSLOduration=2.656435257 podStartE2EDuration="23.690740193s" totalImagesPullingTime="21.034304936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:58.690173108 +0000 UTC m=+24.660901007" watchObservedRunningTime="2026-07-09 18:03:58.690740193 +0000 UTC m=+24.661468091"
Jul 09 18:03:59.558018 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.557988    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:03:59.558018 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.557995    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:03:59.558241 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:59.558111    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:03:59.558241 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:03:59.558188    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:03:59.682322 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.682199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs" event={"ID":"7a3c4b01-1484-4ddf-9f18-5c0d0b24462a","Type":"ContainerStarted","Data":"59a97f04e3b0e2a531f2e3be79fd050f738270bf48b015f352877672abf017d5"}
Jul 09 18:03:59.707716 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.707663    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dqsbs" podStartSLOduration=3.672571868 podStartE2EDuration="24.707644304s" totalImagesPullingTime="21.035072436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:59.707299042 +0000 UTC m=+25.678026942" watchObservedRunningTime="2026-07-09 18:03:59.707644304 +0000 UTC m=+25.678372204"
Jul 09 18:03:59.708366 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.708331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-14.ec2.internal" podStartSLOduration=23.708324548 podStartE2EDuration="23.708324548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:58.707164 +0000 UTC m=+24.677891900" watchObservedRunningTime="2026-07-09 18:03:59.708324548 +0000 UTC m=+25.679052449"
Jul 09 18:03:59.841445 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.841408    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:03:59.841617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.841538    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:03:59.842000 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:03:59.841977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-96g7t"
Jul 09 18:04:00.557754 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:00.557716    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:00.557938 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:00.557861    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:04:01.558379 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:01.558344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:01.558797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:01.558362    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:01.558797 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:01.558472    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:04:01.558797 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:01.558569    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:04:02.561565 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:02.561489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:02.562038 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:02.561588    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:04:02.690015 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:02.689968    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec79ba9f-c9c8-4c31-911f-85e59598901f" containerID="e8b6e1fc0290a21e0316f6c94eded000539bc8de56498de7766979b936aad8af" exitCode=0
Jul 09 18:04:02.690237 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:02.690049    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerDied","Data":"e8b6e1fc0290a21e0316f6c94eded000539bc8de56498de7766979b936aad8af"}
Jul 09 18:04:02.693342 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:02.693326    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:04:02.693666 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:02.693646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"63d9c481bd6413d54b429127ad52497fc634c98cc92ae9062f099d1ddeab0d32"}
Jul 09 18:04:03.558283 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:03.558229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:03.558487 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:03.558270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:03.558487 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:03.558364    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:04:03.558487 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:03.558423    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:04:04.563312 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.561271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:04.563312 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:04.561654    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:04:04.700395 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.700363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:04:04.700759 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.700718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"9531691bb9bdf8c1f7e4b12a5d0c060ffb95a1c3b712cc80ce7644809e3b5e81"}
Jul 09 18:04:04.701003 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.700982    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:04:04.701190 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.701174    2562 scope.go:117] "RemoveContainer" containerID="4f160486d5f34dde1a1a151a2a9558f8e7b2830cf6dbee7e9587368de6d84cc3"
Jul 09 18:04:04.702868 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.702850    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec79ba9f-c9c8-4c31-911f-85e59598901f" containerID="7b28115fdc833f7276cb418c4f4060201bd2d0ede006021973652fcf0a6aebb9" exitCode=0
Jul 09 18:04:04.702966 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.702888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerDied","Data":"7b28115fdc833f7276cb418c4f4060201bd2d0ede006021973652fcf0a6aebb9"}
Jul 09 18:04:04.719796 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:04.719774    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:04:05.557980 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.557954    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:05.558159 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.557994    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:05.558159 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:05.558112    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:04:05.558269 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:05.558188    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:04:05.708510 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.708484    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:04:05.708975 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.708804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" event={"ID":"3502d811-6ad3-4ac5-a954-b170184e50e4","Type":"ContainerStarted","Data":"9a57cd09a1075be1352d43c5189292ff6d3dec83fca13e3f1bfdb7771878d01d"}
Jul 09 18:04:05.709155 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.709130    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:04:05.709263 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.709211    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:04:05.725928 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.725906    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:04:05.771437 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:05.771384    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm" podStartSLOduration=9.698336626 podStartE2EDuration="30.771368884s" totalImagesPullingTime="21.073032258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:05.771295903 +0000 UTC m=+31.742023815" watchObservedRunningTime="2026-07-09 18:04:05.771368884 +0000 UTC m=+31.742096784"
Jul 09 18:04:06.416138 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.416073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mb25c"]
Jul 09 18:04:06.416310 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.416246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:06.416380 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:06.416354    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:04:06.418022 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.417997    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-846sw"]
Jul 09 18:04:06.418168 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.418126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:06.418256 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:06.418237    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:04:06.422515 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.422492    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vbhk9"]
Jul 09 18:04:06.422638 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.422575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:06.422681 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:06.422653    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:04:06.713160 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.713127    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec79ba9f-c9c8-4c31-911f-85e59598901f" containerID="d5cb6c3604ae7bb9a984adf7e063d67a9ec3c7a7391bd11c9ca853dfd7281f33" exitCode=0
Jul 09 18:04:06.713543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:06.713199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerDied","Data":"d5cb6c3604ae7bb9a984adf7e063d67a9ec3c7a7391bd11c9ca853dfd7281f33"}
Jul 09 18:04:07.558682 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:07.558650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:07.558823 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:07.558650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:07.558823 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:07.558750    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:04:07.558902 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:07.558825    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:04:08.213861 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:08.213829    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:08.214370 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:08.213868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:08.214370 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.213978    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:08.214370 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.213978    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:08.214370 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.214039    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret podName:1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.214022188 +0000 UTC m=+66.184750067 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret") pod "global-pull-secret-syncer-mb25c" (UID: "1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:08.214370 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.214056    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.214047638 +0000 UTC m=+66.184775522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:08.415774 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:08.415681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:08.415942 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.415880    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:08.416016 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.415954    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:08.416016 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.415969    2562 projected.go:196] Error preparing data for projected volume kube-api-access-mlf58 for pod openshift-network-diagnostics/network-check-target-vbhk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:08.416126 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.416037    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58 podName:412937f2-07b0-4abf-b8e2-2206ec74df0c nodeName:}" failed. No retries permitted until 2026-07-09 18:04:40.41601651 +0000 UTC m=+66.386744386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlf58" (UniqueName: "kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58") pod "network-check-target-vbhk9" (UID: "412937f2-07b0-4abf-b8e2-2206ec74df0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:08.561413 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:08.561387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:08.561553 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:08.561486    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-846sw" podUID="80b70a46-af4d-46ce-9ef4-443bac043d9d"
Jul 09 18:04:09.557675 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:09.557645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:09.558076 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:09.557647    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:09.558076 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:09.557744    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mb25c" podUID="1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710"
Jul 09 18:04:09.558076 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:09.557808    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vbhk9" podUID="412937f2-07b0-4abf-b8e2-2206ec74df0c"
Jul 09 18:04:10.307790 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.307768    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-14.ec2.internal" event="NodeReady"
Jul 09 18:04:10.307911 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.307896    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:04:10.349033 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.348999    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66c5b75bfb-27257"]
Jul 09 18:04:10.353514 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.353490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.359582 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.359561    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:04:10.359686 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.359673    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:04:10.359819 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.359801    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:04:10.359861 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.359821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4h9p2\""
Jul 09 18:04:10.366634 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.366617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:04:10.369878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.369853    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66c5b75bfb-27257"]
Jul 09 18:04:10.373045 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.373025    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g8mbn"]
Jul 09 18:04:10.378005 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.377986    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jfxn2"]
Jul 09 18:04:10.378146 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.378130    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.381196 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.381174    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:04:10.381196 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.381187    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k2vdg\""
Jul 09 18:04:10.381463 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.381448    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:04:10.381753 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.381726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.388599 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.388580    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:04:10.388790 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.388776    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:04:10.388958 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.388945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w9k7n\""
Jul 09 18:04:10.389159 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.389139    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:04:10.389332 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.389302    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jfxn2"]
Jul 09 18:04:10.389426 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.389394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g8mbn"]
Jul 09 18:04:10.530495 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-image-registry-private-configuration\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530495 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9644\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-kube-api-access-h9644\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530495 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbee485-1e9f-4125-98e3-c4af2363faa5-tmp-dir\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.530495 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbee485-1e9f-4125-98e3-c4af2363faa5-config-volume\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.530733 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-certificates\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530733 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2cr\" (UniqueName: \"kubernetes.io/projected/0fbee485-1e9f-4125-98e3-c4af2363faa5-kube-api-access-ck2cr\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.530733 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a07e1f6-d288-4d69-bab9-05bd8087a18a-cert\") pod \"ingress-canary-jfxn2\" (UID: \"8a07e1f6-d288-4d69-bab9-05bd8087a18a\") " pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.530733 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-trusted-ca\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530733 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwgk\" (UniqueName: \"kubernetes.io/projected/8a07e1f6-d288-4d69-bab9-05bd8087a18a-kube-api-access-7xwgk\") pod \"ingress-canary-jfxn2\" (UID: \"8a07e1f6-d288-4d69-bab9-05bd8087a18a\") " pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.530733 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-installation-pull-secrets\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530892 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530892 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-bound-sa-token\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530892 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff49e4bf-776a-4831-a62a-1e1322b98355-ca-trust-extracted\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.530892 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.530860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.558761 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.558730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:10.566349 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.566314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:10.570367 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.570349    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hfts2\""
Jul 09 18:04:10.631201 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2cr\" (UniqueName: \"kubernetes.io/projected/0fbee485-1e9f-4125-98e3-c4af2363faa5-kube-api-access-ck2cr\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.631357 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a07e1f6-d288-4d69-bab9-05bd8087a18a-cert\") pod \"ingress-canary-jfxn2\" (UID: \"8a07e1f6-d288-4d69-bab9-05bd8087a18a\") " pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.631357 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-trusted-ca\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631357 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwgk\" (UniqueName: \"kubernetes.io/projected/8a07e1f6-d288-4d69-bab9-05bd8087a18a-kube-api-access-7xwgk\") pod \"ingress-canary-jfxn2\" (UID: \"8a07e1f6-d288-4d69-bab9-05bd8087a18a\") " pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.631357 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-installation-pull-secrets\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631357 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-bound-sa-token\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff49e4bf-776a-4831-a62a-1e1322b98355-ca-trust-extracted\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-image-registry-private-configuration\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9644\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-kube-api-access-h9644\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbee485-1e9f-4125-98e3-c4af2363faa5-tmp-dir\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbee485-1e9f-4125-98e3-c4af2363faa5-config-volume\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.631605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.631528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-certificates\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.632132 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.632089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-certificates\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.632303 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.632283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbee485-1e9f-4125-98e3-c4af2363faa5-tmp-dir\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.632662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbee485-1e9f-4125-98e3-c4af2363faa5-config-volume\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:10.632757    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:10.632773    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.632790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff49e4bf-776a-4831-a62a-1e1322b98355-ca-trust-extracted\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:10.632798    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:10.632822    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:11.132806188 +0000 UTC m=+37.103534065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:10.632899    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:11.132872217 +0000 UTC m=+37.103600097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:10.633023 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.632936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-trusted-ca\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.636468 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.636449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-installation-pull-secrets\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.636534 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.636472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a07e1f6-d288-4d69-bab9-05bd8087a18a-cert\") pod \"ingress-canary-jfxn2\" (UID: \"8a07e1f6-d288-4d69-bab9-05bd8087a18a\") " pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.636762 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.636740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-image-registry-private-configuration\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.642409 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.642387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9644\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-kube-api-access-h9644\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.642844 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.642823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-bound-sa-token\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:10.642979 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.642959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwgk\" (UniqueName: \"kubernetes.io/projected/8a07e1f6-d288-4d69-bab9-05bd8087a18a-kube-api-access-7xwgk\") pod \"ingress-canary-jfxn2\" (UID: \"8a07e1f6-d288-4d69-bab9-05bd8087a18a\") " pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.643129 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.643086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2cr\" (UniqueName: \"kubernetes.io/projected/0fbee485-1e9f-4125-98e3-c4af2363faa5-kube-api-access-ck2cr\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:10.696500 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.696467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jfxn2"
Jul 09 18:04:10.853531 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:10.853287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jfxn2"]
Jul 09 18:04:11.135901 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.135780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:11.135901 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.135874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:11.136148 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:11.135938    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:11.136148 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:11.135966    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:11.136148 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:11.135979    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:11.136148 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:11.136010    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:12.135986438 +0000 UTC m=+38.106714315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:11.136148 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:11.136032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:12.136021661 +0000 UTC m=+38.106749538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:11.558077 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.558036    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:11.558251 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.558036    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:11.561419 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.561396    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:04:11.562775 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.562755    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:04:11.562885 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.562773    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vlll4\""
Jul 09 18:04:11.562885 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.562798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:11.681476 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:11.681432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:12.144605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:12.144562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:12.144782 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:12.144702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:12.144782 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:12.144710    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:12.144782 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:12.144781    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:14.144762338 +0000 UTC m=+40.115490238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:12.144899 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:12.144791    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:12.144899 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:12.144802    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:12.144899 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:12.144833    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:14.144822896 +0000 UTC m=+40.115550772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:12.159116 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:12.159067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:12.344453 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:04:12.344413    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a07e1f6_d288_4d69_bab9_05bd8087a18a.slice/crio-450cba07ba0d798c9afb076b590648dbbdcb9e1afedaa4a5b402e202d29cefc3 WatchSource:0}: Error finding container 450cba07ba0d798c9afb076b590648dbbdcb9e1afedaa4a5b402e202d29cefc3: Status 404 returned error can't find the container with id 450cba07ba0d798c9afb076b590648dbbdcb9e1afedaa4a5b402e202d29cefc3
Jul 09 18:04:12.733280 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:12.733187    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec79ba9f-c9c8-4c31-911f-85e59598901f" containerID="7412b517ab1877b2445f344614e28eec36dafc1a34f629b9c966c7d88ec2aa2e" exitCode=0
Jul 09 18:04:12.733664 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:12.733275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerDied","Data":"7412b517ab1877b2445f344614e28eec36dafc1a34f629b9c966c7d88ec2aa2e"}
Jul 09 18:04:12.734367 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:12.734349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jfxn2" event={"ID":"8a07e1f6-d288-4d69-bab9-05bd8087a18a","Type":"ContainerStarted","Data":"450cba07ba0d798c9afb076b590648dbbdcb9e1afedaa4a5b402e202d29cefc3"}
Jul 09 18:04:13.744061 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:13.744021    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec79ba9f-c9c8-4c31-911f-85e59598901f" containerID="eaf9d02aa8bf210371bf121832356491201d642b292954768d4c5804ef619e0e" exitCode=0
Jul 09 18:04:13.744594 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:13.744082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerDied","Data":"eaf9d02aa8bf210371bf121832356491201d642b292954768d4c5804ef619e0e"}
Jul 09 18:04:14.159659 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:14.159625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:14.159845 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:14.159688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:14.159845 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:14.159739    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:14.159845 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:14.159756    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:14.159845 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:14.159777    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:14.159845 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:14.159818    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.159800183 +0000 UTC m=+44.130528060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:14.159845 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:14.159833    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:18.159827528 +0000 UTC m=+44.130555404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:14.751553 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:14.751516    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" event={"ID":"ec79ba9f-c9c8-4c31-911f-85e59598901f","Type":"ContainerStarted","Data":"886c8e47a4828fea5b00766919c2eaf8878d559fce4d29e3458386e6f1ac8f47"}
Jul 09 18:04:14.778866 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:14.778817    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7xqwq" podStartSLOduration=11.745987342 podStartE2EDuration="39.778801719s" totalImagesPullingTime="28.032814377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:14.778445614 +0000 UTC m=+40.749173512" watchObservedRunningTime="2026-07-09 18:04:14.778801719 +0000 UTC m=+40.749529651"
Jul 09 18:04:14.915183 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:14.915076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:15.015331 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:15.015250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:15.755790 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:15.755557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jfxn2" event={"ID":"8a07e1f6-d288-4d69-bab9-05bd8087a18a","Type":"ContainerStarted","Data":"a42bfa4627b5a490dee39495c1f22988988e8cde0091cc46c86879601bc3792b"}
Jul 09 18:04:15.773522 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:15.773478    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jfxn2" podStartSLOduration=2.6844622510000002 podStartE2EDuration="5.773463034s" totalImagesPullingTime="3.089000783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:15.773410585 +0000 UTC m=+41.744138483" watchObservedRunningTime="2026-07-09 18:04:15.773463034 +0000 UTC m=+41.744190961"
Jul 09 18:04:18.188693 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:18.188652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:18.189313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:18.188755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:18.189313 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:18.188828    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:18.189313 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:18.188848    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:18.189313 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:18.188858    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:18.189313 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:18.188908    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.188887542 +0000 UTC m=+52.159615436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:18.189313 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:18.188926    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:26.188918291 +0000 UTC m=+52.159646168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:18.936031 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:18.935992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:20.760006 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:20.759974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:22.783058 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:22.783017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:22.891177 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:22.891144    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:23.061407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:23.061325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:23.160899 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:23.160854    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:26.249919 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:26.249889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:26.250432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:26.249927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:26.250432 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:26.250019    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:26.250432 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:26.250044    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:26.250432 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:26.250061    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:26.250432 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:26.250072    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.25005764 +0000 UTC m=+68.220785517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:26.250432 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:26.250117    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:42.250089367 +0000 UTC m=+68.220817244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:37.726715 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:37.726688    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrngm"
Jul 09 18:04:37.998230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:37.998140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:38.155663 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:38.155631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:38.299250 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:38.299158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:40.245928 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.245890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:04:40.246390 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.245939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:40.249328 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.249313    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:04:40.249629 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.249613    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:04:40.256953 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:40.256932    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:04:40.257050 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:40.256990    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs podName:80b70a46-af4d-46ce-9ef4-443bac043d9d nodeName:}" failed. No retries permitted until 2026-07-09 18:05:44.25697357 +0000 UTC m=+130.227701447 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs") pod "network-metrics-daemon-846sw" (UID: "80b70a46-af4d-46ce-9ef4-443bac043d9d") : secret "metrics-daemon-secret" not found
Jul 09 18:04:40.260882 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.260863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710-original-pull-secret\") pod \"global-pull-secret-syncer-mb25c\" (UID: \"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710\") " pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:40.370932 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.370882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mb25c"
Jul 09 18:04:40.447534 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.447496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:40.450939 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.450915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:04:40.461274 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.461250    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:04:40.471415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.471387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlf58\" (UniqueName: \"kubernetes.io/projected/412937f2-07b0-4abf-b8e2-2206ec74df0c-kube-api-access-mlf58\") pod \"network-check-target-vbhk9\" (UID: \"412937f2-07b0-4abf-b8e2-2206ec74df0c\") " pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:40.503279 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.503218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mb25c"]
Jul 09 18:04:40.506376 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:04:40.506354    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb5fe9a_16d5_4bb5_a4cc_f62027c9b710.slice/crio-dd848190a9a4ec6823baa1dcb4f2aae13e68f9114c93fbe0fb9952cdca07599f WatchSource:0}: Error finding container dd848190a9a4ec6823baa1dcb4f2aae13e68f9114c93fbe0fb9952cdca07599f: Status 404 returned error can't find the container with id dd848190a9a4ec6823baa1dcb4f2aae13e68f9114c93fbe0fb9952cdca07599f
Jul 09 18:04:40.680256 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.680223    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vlll4\""
Jul 09 18:04:40.686985 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.686958    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:40.813895 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.813864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mb25c" event={"ID":"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710","Type":"ContainerStarted","Data":"dd848190a9a4ec6823baa1dcb4f2aae13e68f9114c93fbe0fb9952cdca07599f"}
Jul 09 18:04:40.821147 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.821121    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vbhk9"]
Jul 09 18:04:40.824057 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:04:40.824032    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412937f2_07b0_4abf_b8e2_2206ec74df0c.slice/crio-4a9bb4ca2449f3d4ad4e977cbb797063e66f47898c91188a2964dae800d279f2 WatchSource:0}: Error finding container 4a9bb4ca2449f3d4ad4e977cbb797063e66f47898c91188a2964dae800d279f2: Status 404 returned error can't find the container with id 4a9bb4ca2449f3d4ad4e977cbb797063e66f47898c91188a2964dae800d279f2
Jul 09 18:04:40.957667 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.957633    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"]
Jul 09 18:04:40.960392 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.960375    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:40.969182 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.969157    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"]
Jul 09 18:04:40.971288 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.971272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:40.976037 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.976018    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 18:04:40.976037 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.976027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 18:04:40.976196 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.976027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rpx2p\""
Jul 09 18:04:40.976878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.976861    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 18:04:40.977076 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.977054    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:04:40.977076 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.977068    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 18:04:40.977251 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.977054    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:04:40.977251 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.977184    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:04:40.981878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.981831    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"]
Jul 09 18:04:40.992132 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.992090    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"]
Jul 09 18:04:40.994220 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.994205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:40.997360 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:40.997342    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:04:41.003582 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.003564    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:04:41.014710 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.014652    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"]
Jul 09 18:04:41.015444 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.015420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"]
Jul 09 18:04:41.052312 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a59029e-f8b0-4b50-ace8-8c975dbf1660-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b596b46b4-lvk6b\" (UID: \"3a59029e-f8b0-4b50-ace8-8c975dbf1660\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.052312 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t829j\" (UniqueName: \"kubernetes.io/projected/3a59029e-f8b0-4b50-ace8-8c975dbf1660-kube-api-access-t829j\") pod \"managed-serviceaccount-addon-agent-6b596b46b4-lvk6b\" (UID: \"3a59029e-f8b0-4b50-ace8-8c975dbf1660\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-hub\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb4c4c38-8552-4177-9b6c-e6e348de2e24-tmp\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eb4c4c38-8552-4177-9b6c-e6e348de2e24-klusterlet-config\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgnm\" (UniqueName: \"kubernetes.io/projected/eb4c4c38-8552-4177-9b6c-e6e348de2e24-kube-api-access-9tgnm\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-ca\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.052470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.052675 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bq6\" (UniqueName: \"kubernetes.io/projected/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-kube-api-access-b9bq6\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.052675 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.052531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153291 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-ca\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153291 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bq6\" (UniqueName: \"kubernetes.io/projected/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-kube-api-access-b9bq6\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a59029e-f8b0-4b50-ace8-8c975dbf1660-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b596b46b4-lvk6b\" (UID: \"3a59029e-f8b0-4b50-ace8-8c975dbf1660\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t829j\" (UniqueName: \"kubernetes.io/projected/3a59029e-f8b0-4b50-ace8-8c975dbf1660-kube-api-access-t829j\") pod \"managed-serviceaccount-addon-agent-6b596b46b4-lvk6b\" (UID: \"3a59029e-f8b0-4b50-ace8-8c975dbf1660\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-hub\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb4c4c38-8552-4177-9b6c-e6e348de2e24-tmp\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eb4c4c38-8552-4177-9b6c-e6e348de2e24-klusterlet-config\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.153543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.153529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgnm\" (UniqueName: \"kubernetes.io/projected/eb4c4c38-8552-4177-9b6c-e6e348de2e24-kube-api-access-9tgnm\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.154172 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.154144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb4c4c38-8552-4177-9b6c-e6e348de2e24-tmp\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.154442 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.154276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.156244 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.156218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.156345 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.156311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-ca\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.156549 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.156531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.156751 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.156733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-hub\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.156751 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.156745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eb4c4c38-8552-4177-9b6c-e6e348de2e24-klusterlet-config\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.156930 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.156912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a59029e-f8b0-4b50-ace8-8c975dbf1660-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b596b46b4-lvk6b\" (UID: \"3a59029e-f8b0-4b50-ace8-8c975dbf1660\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.175320 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.175296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgnm\" (UniqueName: \"kubernetes.io/projected/eb4c4c38-8552-4177-9b6c-e6e348de2e24-kube-api-access-9tgnm\") pod \"klusterlet-addon-workmgr-68949578c7-4xrvm\" (UID: \"eb4c4c38-8552-4177-9b6c-e6e348de2e24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.178705 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.178683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t829j\" (UniqueName: \"kubernetes.io/projected/3a59029e-f8b0-4b50-ace8-8c975dbf1660-kube-api-access-t829j\") pod \"managed-serviceaccount-addon-agent-6b596b46b4-lvk6b\" (UID: \"3a59029e-f8b0-4b50-ace8-8c975dbf1660\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.191720 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.191698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bq6\" (UniqueName: \"kubernetes.io/projected/6a5a3065-70ae-4c34-9e54-f65bb7c7f263-kube-api-access-b9bq6\") pod \"cluster-proxy-proxy-agent-8fb6744bb-jdl9v\" (UID: \"6a5a3065-70ae-4c34-9e54-f65bb7c7f263\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.278078 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.277993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"
Jul 09 18:04:41.285085 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.285063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:04:41.311754 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.311728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:41.461906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.461882    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b"]
Jul 09 18:04:41.462012 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:04:41.461918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a59029e_f8b0_4b50_ace8_8c975dbf1660.slice/crio-9e3cb0107ac40fb211ecf315e00e5d895bba499c6786f19a16c26d28f9201da4 WatchSource:0}: Error finding container 9e3cb0107ac40fb211ecf315e00e5d895bba499c6786f19a16c26d28f9201da4: Status 404 returned error can't find the container with id 9e3cb0107ac40fb211ecf315e00e5d895bba499c6786f19a16c26d28f9201da4
Jul 09 18:04:41.470061 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.470037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"]
Jul 09 18:04:41.471759 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:04:41.471724    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a5a3065_70ae_4c34_9e54_f65bb7c7f263.slice/crio-27898704836bf6a09294caf61acf101c5feb63c7f6bc5f705e4c4329b06342b3 WatchSource:0}: Error finding container 27898704836bf6a09294caf61acf101c5feb63c7f6bc5f705e4c4329b06342b3: Status 404 returned error can't find the container with id 27898704836bf6a09294caf61acf101c5feb63c7f6bc5f705e4c4329b06342b3
Jul 09 18:04:41.495617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.495592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"]
Jul 09 18:04:41.565575 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.565496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:41.696972 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.696830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:41.818325 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.818238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b" event={"ID":"3a59029e-f8b0-4b50-ace8-8c975dbf1660","Type":"ContainerStarted","Data":"9e3cb0107ac40fb211ecf315e00e5d895bba499c6786f19a16c26d28f9201da4"}
Jul 09 18:04:41.819827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.819799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm" event={"ID":"eb4c4c38-8552-4177-9b6c-e6e348de2e24","Type":"ContainerStarted","Data":"765c73cc57279f4c577e7b9926cae48916f8354a3ffcf2a453ef88804609e10b"}
Jul 09 18:04:41.821148 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.821081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" event={"ID":"6a5a3065-70ae-4c34-9e54-f65bb7c7f263","Type":"ContainerStarted","Data":"27898704836bf6a09294caf61acf101c5feb63c7f6bc5f705e4c4329b06342b3"}
Jul 09 18:04:41.822389 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.822367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vbhk9" event={"ID":"412937f2-07b0-4abf-b8e2-2206ec74df0c","Type":"ContainerStarted","Data":"4a9bb4ca2449f3d4ad4e977cbb797063e66f47898c91188a2964dae800d279f2"}
Jul 09 18:04:41.867888 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:41.867858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:42.264216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:42.264280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:42.264482    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:42.264548    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls podName:0fbee485-1e9f-4125-98e3-c4af2363faa5 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:14.264529143 +0000 UTC m=+100.235257024 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls") pod "dns-default-g8mbn" (UID: "0fbee485-1e9f-4125-98e3-c4af2363faa5") : secret "dns-default-metrics-tls" not found
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:42.264945    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:42.264961    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66c5b75bfb-27257: secret "image-registry-tls" not found
Jul 09 18:04:42.265040 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:04:42.265008    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls podName:ff49e4bf-776a-4831-a62a-1e1322b98355 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:14.264993155 +0000 UTC m=+100.235721035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls") pod "image-registry-66c5b75bfb-27257" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355") : secret "image-registry-tls" not found
Jul 09 18:04:42.366120 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:42.366055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:42.487318 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:42.487173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:44.398078 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:44.397920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:44.523936 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:44.523903    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:44.653435 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:44.653343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.065739 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:45.065649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.424853 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:45.424799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:45.532235 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:45.532197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:49.748912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:49.748876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:50.851171 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.851060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm" event={"ID":"eb4c4c38-8552-4177-9b6c-e6e348de2e24","Type":"ContainerStarted","Data":"2e684922cb3bb1c38bc0986e7c027a414b1deb6d92784fdbcbb64fdfe4118f49"}
Jul 09 18:04:50.851602 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.851332    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:50.852791 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.852765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mb25c" event={"ID":"1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710","Type":"ContainerStarted","Data":"a9b10085f2a4e581e1505a6e7179aefd98a934b192d996e3b77a3c7e880b0369"}
Jul 09 18:04:50.853106 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.853069    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm"
Jul 09 18:04:50.854765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.854745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" event={"ID":"6a5a3065-70ae-4c34-9e54-f65bb7c7f263","Type":"ContainerStarted","Data":"f1aee1f9e74cbe81c38df5fa364211e6e8c370c039ebbfb4ae8e461955168a6c"}
Jul 09 18:04:50.854873 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.854778    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" event={"ID":"6a5a3065-70ae-4c34-9e54-f65bb7c7f263","Type":"ContainerStarted","Data":"e5d8b03f34ae83e158df33d998bc0b04c53fae642d8a8d4de3cbabeae4141a23"}
Jul 09 18:04:50.854873 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.854795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" event={"ID":"6a5a3065-70ae-4c34-9e54-f65bb7c7f263","Type":"ContainerStarted","Data":"a6e0bf175b83fb7787f86eebee192e0a8022ebb7d64670c25c09272089420c49"}
Jul 09 18:04:50.856273 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.856254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vbhk9" event={"ID":"412937f2-07b0-4abf-b8e2-2206ec74df0c","Type":"ContainerStarted","Data":"c7595aa7d4666afdf7405c6f7e6811df6c40ac532ed51e44c742997950ef733a"}
Jul 09 18:04:50.856356 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.856303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:04:50.857524 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.857501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b" event={"ID":"3a59029e-f8b0-4b50-ace8-8c975dbf1660","Type":"ContainerStarted","Data":"29e0d9c33898cd122fd4aaeb7d472ee7d9cab7a207d280aaaaa39053c8fada11"}
Jul 09 18:04:50.869339 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.869299    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68949578c7-4xrvm" podStartSLOduration=2.111547897 podStartE2EDuration="10.869284382s" totalImagesPullingTime="8.757736485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.868500662 +0000 UTC m=+76.839228561" watchObservedRunningTime="2026-07-09 18:04:50.869284382 +0000 UTC m=+76.840012280"
Jul 09 18:04:50.888207 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.888161    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mb25c" podStartSLOduration=66.156782982 podStartE2EDuration="1m15.888140784s" totalImagesPullingTime="9.731357802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.887765215 +0000 UTC m=+76.858493117" watchObservedRunningTime="2026-07-09 18:04:50.888140784 +0000 UTC m=+76.858868722"
Jul 09 18:04:50.903652 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.903602    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vbhk9" podStartSLOduration=66.494149475 podStartE2EDuration="1m15.903587259s" totalImagesPullingTime="9.409437784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.903202688 +0000 UTC m=+76.873930587" watchObservedRunningTime="2026-07-09 18:04:50.903587259 +0000 UTC m=+76.874315157"
Jul 09 18:04:50.937242 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.937193    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b596b46b4-lvk6b" podStartSLOduration=2.169336439 podStartE2EDuration="10.937178295s" totalImagesPullingTime="8.767841856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.937019548 +0000 UTC m=+76.907747448" watchObservedRunningTime="2026-07-09 18:04:50.937178295 +0000 UTC m=+76.907906195"
Jul 09 18:04:50.958870 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:50.958826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" podStartSLOduration=2.184838497 podStartE2EDuration="10.958812927s" totalImagesPullingTime="8.77397443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:50.958264366 +0000 UTC m=+76.928992278" watchObservedRunningTime="2026-07-09 18:04:50.958812927 +0000 UTC m=+76.929540834"
Jul 09 18:04:54.318570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:54.318537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:54.422230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:54.422203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.408489 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:56.408376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.509719 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:56.509683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.614047 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:56.614012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:56.718139 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:56.718034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:58.157279 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:58.157246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:58.260958 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:58.260921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:58.358928 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:58.358895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:04:58.467989 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:04:58.467901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:01.389725 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:01.389690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:01.508492 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:01.508457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:01.662211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:01.662123    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:01.821109 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:01.821058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:02.003581 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:02.003494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:02.093163 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:02.093133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:02.427208 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:02.427168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:02.550328 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:02.550299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.095160 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:04.095130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.238033 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:04.238001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.537788 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:04.537752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:04.642333 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:04.642302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:05.623974 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:05.623934    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:06.085414 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:06.085372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:06.233158 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:06.233125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:08.701192 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:08.701157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:08.720600 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:08.720578    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j45n7_11a66269-f70d-45a4-9bf4-0599903b8af8/dns-node-resolver/0.log"
Jul 09 18:05:08.819840 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:08.819807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:08.966020 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:08.965935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:09.130468 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:09.130436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:09.248558 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:09.248466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:09.372333 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:09.372304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:09.714709 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:09.714689    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvsm2_13cb5c00-60d1-4a33-8302-cac1198f9c1f/node-ca/0.log"
Jul 09 18:05:10.179193 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:10.179162    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:10.373246 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:10.373213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:10.519049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:10.518972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jfxn2_8a07e1f6-d288-4d69-bab9-05bd8087a18a/serve-healthcheck-canary/0.log"
Jul 09 18:05:10.676989 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:10.676952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:10.873068 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:10.872982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:11.508554 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:11.508514    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:11.899021 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:11.898986    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.276900 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:12.276818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.669723 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:12.669687    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.764299 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:12.764268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.878280 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:12.878250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:12.998345 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:12.998255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.253516 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:13.253435    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.347577 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:13.347540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:13.670827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:13.670793    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.289059 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.289026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.309007 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.308980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:05:14.309125 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.309014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:05:14.311532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.311504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0fbee485-1e9f-4125-98e3-c4af2363faa5-metrics-tls\") pod \"dns-default-g8mbn\" (UID: \"0fbee485-1e9f-4125-98e3-c4af2363faa5\") " pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:05:14.311648 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.311559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"image-registry-66c5b75bfb-27257\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") " pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:05:14.568013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.567934    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4h9p2\""
Jul 09 18:05:14.575403 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.575386    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:05:14.593626 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.593603    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k2vdg\""
Jul 09 18:05:14.599270 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.599244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:05:14.735487 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.735454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66c5b75bfb-27257"]
Jul 09 18:05:14.738232 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:05:14.738186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff49e4bf_776a_4831_a62a_1e1322b98355.slice/crio-a46082cd305c754453f9d4f7411fdf4973f0a09021c5f0173245cf2f351f736e WatchSource:0}: Error finding container a46082cd305c754453f9d4f7411fdf4973f0a09021c5f0173245cf2f351f736e: Status 404 returned error can't find the container with id a46082cd305c754453f9d4f7411fdf4973f0a09021c5f0173245cf2f351f736e
Jul 09 18:05:14.757432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.757366    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g8mbn"]
Jul 09 18:05:14.766089 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:05:14.766065    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbee485_1e9f_4125_98e3_c4af2363faa5.slice/crio-5bca70a684190352d662c7d2673e870f16ce132873f06eb1873d8defa9c4810d WatchSource:0}: Error finding container 5bca70a684190352d662c7d2673e870f16ce132873f06eb1873d8defa9c4810d: Status 404 returned error can't find the container with id 5bca70a684190352d662c7d2673e870f16ce132873f06eb1873d8defa9c4810d
Jul 09 18:05:14.859379 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.859309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:14.928157 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.928128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g8mbn" event={"ID":"0fbee485-1e9f-4125-98e3-c4af2363faa5","Type":"ContainerStarted","Data":"5bca70a684190352d662c7d2673e870f16ce132873f06eb1873d8defa9c4810d"}
Jul 09 18:05:14.929524 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.929497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" event={"ID":"ff49e4bf-776a-4831-a62a-1e1322b98355","Type":"ContainerStarted","Data":"68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8"}
Jul 09 18:05:14.929650 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.929535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" event={"ID":"ff49e4bf-776a-4831-a62a-1e1322b98355","Type":"ContainerStarted","Data":"a46082cd305c754453f9d4f7411fdf4973f0a09021c5f0173245cf2f351f736e"}
Jul 09 18:05:14.929650 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.929631    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:05:14.951920 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.951881    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" podStartSLOduration=99.951866944 podStartE2EDuration="1m39.951866944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:14.951143487 +0000 UTC m=+100.921871386" watchObservedRunningTime="2026-07-09 18:05:14.951866944 +0000 UTC m=+100.922594820"
Jul 09 18:05:14.963299 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:14.963269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.093280 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:15.093241    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.220167 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:15.220134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.414953 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:15.414914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:15.731940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:15.731897    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:16.817619 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:16.817585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:16.917522 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:16.917492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.024247 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.024214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.137532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.137501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.853738 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.853643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.940930 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.940901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g8mbn" event={"ID":"0fbee485-1e9f-4125-98e3-c4af2363faa5","Type":"ContainerStarted","Data":"4ffaa23a9b2fca81075a2f5c54f30c5eab65d25b3efbaef2e8250f65a162371a"}
Jul 09 18:05:17.941119 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.940958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g8mbn" event={"ID":"0fbee485-1e9f-4125-98e3-c4af2363faa5","Type":"ContainerStarted","Data":"631ab65f0f710039bd9cdde8bb29fdc5aec168a522fe6437d01c9ca5ce75d847"}
Jul 09 18:05:17.941119 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.941011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:05:17.947885 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.947857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:17.965141 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:17.965086    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g8mbn" podStartSLOduration=65.583150827 podStartE2EDuration="1m7.96507449s" totalImagesPullingTime="2.381923663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:17.963293383 +0000 UTC m=+103.934021281" watchObservedRunningTime="2026-07-09 18:05:17.96507449 +0000 UTC m=+103.935802388"
Jul 09 18:05:18.244576 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:18.244529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.661167 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:18.661129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:18.852449 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:18.852416    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.266561 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:19.266527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:19.496415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:19.496379    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:21.863031 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:21.863001    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vbhk9"
Jul 09 18:05:26.599083 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:26.599039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:26.693215 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:26.693183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:27.946162 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:27.946139    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g8mbn"
Jul 09 18:05:28.023676 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.023646    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xbfxf"]
Jul 09 18:05:28.025867 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.025852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.030955 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.030936    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tcgvj\""
Jul 09 18:05:28.032192 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.032169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:05:28.032295 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.032204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:05:28.032295 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.032204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:05:28.032593 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.032575    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:05:28.056631 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.056610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbfxf"]
Jul 09 18:05:28.109107 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.109062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-data-volume\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.109242 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.109111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.109293 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.109262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-crio-socket\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.109344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.109302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdbm\" (UniqueName: \"kubernetes.io/projected/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-kube-api-access-4zdbm\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.109344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.109335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.145415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.145391    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.210769 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.210674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-data-volume\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.210769 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.210718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.210940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.210858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-crio-socket\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.210940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.210891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdbm\" (UniqueName: \"kubernetes.io/projected/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-kube-api-access-4zdbm\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.210940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.210918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.211033 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.210980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-crio-socket\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.211065 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.211052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-data-volume\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.211396 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.211380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.213305 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.213285    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.243201 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.243173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdbm\" (UniqueName: \"kubernetes.io/projected/c835b3ab-3208-4ee4-ae39-5ebc772e92e4-kube-api-access-4zdbm\") pod \"insights-runtime-extractor-xbfxf\" (UID: \"c835b3ab-3208-4ee4-ae39-5ebc772e92e4\") " pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.273519 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.273489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.335007 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.334977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbfxf"
Jul 09 18:05:28.374582 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.374552    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.479515 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.479492    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbfxf"]
Jul 09 18:05:28.480509 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.480483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.482241 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:05:28.482219    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc835b3ab_3208_4ee4_ae39_5ebc772e92e4.slice/crio-e52caf938b5e02ecf5649c8b45151e61e0cbd7df58cd7ead56dda5b5af7e8a4e WatchSource:0}: Error finding container e52caf938b5e02ecf5649c8b45151e61e0cbd7df58cd7ead56dda5b5af7e8a4e: Status 404 returned error can't find the container with id e52caf938b5e02ecf5649c8b45151e61e0cbd7df58cd7ead56dda5b5af7e8a4e
Jul 09 18:05:28.570806 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.570775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:28.980371 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:28.980254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbfxf" event={"ID":"c835b3ab-3208-4ee4-ae39-5ebc772e92e4","Type":"ContainerStarted","Data":"e52caf938b5e02ecf5649c8b45151e61e0cbd7df58cd7ead56dda5b5af7e8a4e"}
Jul 09 18:05:29.401608 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:29.401518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.776358 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:29.776267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:29.878767 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:29.878728    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:32.992755 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:32.992724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbfxf" event={"ID":"c835b3ab-3208-4ee4-ae39-5ebc772e92e4","Type":"ContainerStarted","Data":"dae244c7945ceb956b988477808dca259adb1f3564ab5161286494c2a86d228f"}
Jul 09 18:05:32.993069 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:32.992788    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbfxf" event={"ID":"c835b3ab-3208-4ee4-ae39-5ebc772e92e4","Type":"ContainerStarted","Data":"aca63f756435a5927f2a963b12c2805ce06080c71e3cb1a0440f243a6cacf1c2"}
Jul 09 18:05:34.581355 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:34.581307    2562 patch_prober.go:28] interesting pod/image-registry-66c5b75bfb-27257 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:05:34.581785 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:34.581374    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" podUID="ff49e4bf-776a-4831-a62a-1e1322b98355" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:05:35.466312 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:35.466284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:35.564820 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:35.564778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:35.938532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:35.938510    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:05:36.006378 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:36.006344    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbfxf" event={"ID":"c835b3ab-3208-4ee4-ae39-5ebc772e92e4","Type":"ContainerStarted","Data":"73194fdfa9b6e265f2365a0065560f15cb171d0560cf172430ca5cc2534c338d"}
Jul 09 18:05:36.026666 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:36.026610    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xbfxf" podStartSLOduration=2.178192634 podStartE2EDuration="9.0265944s" totalImagesPullingTime="6.848401766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:36.025949875 +0000 UTC m=+121.996677776" watchObservedRunningTime="2026-07-09 18:05:36.0265944 +0000 UTC m=+121.997322298"
Jul 09 18:05:36.463079 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:36.463042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:36.547301 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:36.547266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.108999 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:37.108965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.288596 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:37.288560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.563552 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:37.563518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:37.651329 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:37.651299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.669332 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:38.669291    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.753472 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:38.753439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:38.832228 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:38.832190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:41.014607 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:41.014572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:42.837724 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:42.837698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:42.960394 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:42.960362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.102419 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.102335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.186932 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.186905    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w54q8"]
Jul 09 18:05:43.190570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.190547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.193420 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.193394    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:05:43.193566 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.193451    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:05:43.193566 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.193469    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:05:43.193783 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.193764    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-92pww\""
Jul 09 18:05:43.194840 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.194819    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:05:43.194840 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.194835    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:05:43.195002 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.194828    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:05:43.212244 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.212223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.315019 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.314984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.327016 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.326993    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-root\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-accelerators-collector-config\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-tls\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-wtmp\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z569t\" (UniqueName: \"kubernetes.io/projected/445f8337-c953-4736-9a22-c89a9d895e61-kube-api-access-z569t\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-textfile\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445f8337-c953-4736-9a22-c89a9d895e61-metrics-client-ca\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.327313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.327259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-sys\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.416497 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.416469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.427625 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427698 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-sys\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427735 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-sys\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427773 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-root\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427806 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-accelerators-collector-config\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427838 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-root\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427838 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-tls\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427898 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-wtmp\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427898 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z569t\" (UniqueName: \"kubernetes.io/projected/445f8337-c953-4736-9a22-c89a9d895e61-kube-api-access-z569t\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-textfile\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.427992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.427938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445f8337-c953-4736-9a22-c89a9d895e61-metrics-client-ca\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.428079 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.428013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-wtmp\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.428364 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.428343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-textfile\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.428465 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.428441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-accelerators-collector-config\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.428686 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.428498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/445f8337-c953-4736-9a22-c89a9d895e61-metrics-client-ca\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.430019 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.429993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.430293 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.430273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/445f8337-c953-4736-9a22-c89a9d895e61-node-exporter-tls\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.441390 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.441363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z569t\" (UniqueName: \"kubernetes.io/projected/445f8337-c953-4736-9a22-c89a9d895e61-kube-api-access-z569t\") pod \"node-exporter-w54q8\" (UID: \"445f8337-c953-4736-9a22-c89a9d895e61\") " pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.501492 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.501458    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w54q8"
Jul 09 18:05:43.512942 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:05:43.512919    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445f8337_c953_4736_9a22_c89a9d895e61.slice/crio-892df6b1b013ae5e3dbe4fdae2171c34379685b1e6cee3a091d31ed6cfcecbb2 WatchSource:0}: Error finding container 892df6b1b013ae5e3dbe4fdae2171c34379685b1e6cee3a091d31ed6cfcecbb2: Status 404 returned error can't find the container with id 892df6b1b013ae5e3dbe4fdae2171c34379685b1e6cee3a091d31ed6cfcecbb2
Jul 09 18:05:43.626373 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.626338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.773980 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.773873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:43.924111 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:43.924058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.037860 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.037771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w54q8" event={"ID":"445f8337-c953-4736-9a22-c89a9d895e61","Type":"ContainerStarted","Data":"892df6b1b013ae5e3dbe4fdae2171c34379685b1e6cee3a091d31ed6cfcecbb2"}
Jul 09 18:05:44.121833 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.121794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.239874 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.239842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.338175 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.338079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:05:44.340954 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.340922    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b70a46-af4d-46ce-9ef4-443bac043d9d-metrics-certs\") pod \"network-metrics-daemon-846sw\" (UID: \"80b70a46-af4d-46ce-9ef4-443bac043d9d\") " pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:05:44.442838 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.442806    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:44.471373 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.471347    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hfts2\""
Jul 09 18:05:44.479551 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.479533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-846sw"
Jul 09 18:05:44.616690 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:44.616609    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-846sw"]
Jul 09 18:05:44.619323 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:05:44.619297    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b70a46_af4d_46ce_9ef4_443bac043d9d.slice/crio-f11372ccce224c39b8f278b6a2dec4d17aeaa9821752a7d43f792f2313680cc0 WatchSource:0}: Error finding container f11372ccce224c39b8f278b6a2dec4d17aeaa9821752a7d43f792f2313680cc0: Status 404 returned error can't find the container with id f11372ccce224c39b8f278b6a2dec4d17aeaa9821752a7d43f792f2313680cc0
Jul 09 18:05:45.042408 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:45.042369    2562 generic.go:356] "Generic (PLEG): container finished" podID="445f8337-c953-4736-9a22-c89a9d895e61" containerID="3d9913a3be22a2305fef8152b6c8e0a4eeec0df40a6c00df2b477f852ef28272" exitCode=0
Jul 09 18:05:45.042863 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:45.042462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w54q8" event={"ID":"445f8337-c953-4736-9a22-c89a9d895e61","Type":"ContainerDied","Data":"3d9913a3be22a2305fef8152b6c8e0a4eeec0df40a6c00df2b477f852ef28272"}
Jul 09 18:05:45.043595 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:45.043573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-846sw" event={"ID":"80b70a46-af4d-46ce-9ef4-443bac043d9d","Type":"ContainerStarted","Data":"f11372ccce224c39b8f278b6a2dec4d17aeaa9821752a7d43f792f2313680cc0"}
Jul 09 18:05:46.049765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.049728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w54q8" event={"ID":"445f8337-c953-4736-9a22-c89a9d895e61","Type":"ContainerStarted","Data":"3303d7098b685abb75bef8d1c0f763252aa1a57aa1286a7ecfb67389c0123b14"}
Jul 09 18:05:46.050255 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.049791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w54q8" event={"ID":"445f8337-c953-4736-9a22-c89a9d895e61","Type":"ContainerStarted","Data":"5c3dad3d57a878c4cb305c15d63feac0a9307024ab8f71f0aeda6134aaaa3105"}
Jul 09 18:05:46.076366 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.076306    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w54q8" podStartSLOduration=2.053437657 podStartE2EDuration="3.076285477s" totalImagesPullingTime="948.226573ms" totalInitContainerRuntime="74.621247ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:46.074483193 +0000 UTC m=+132.045211093" watchObservedRunningTime="2026-07-09 18:05:46.076285477 +0000 UTC m=+132.047013379"
Jul 09 18:05:46.145496 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.145467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.251172 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.251139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.351715 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.351640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.719497 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.719464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:46.818782 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:46.818747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.054705 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:47.054613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-846sw" event={"ID":"80b70a46-af4d-46ce-9ef4-443bac043d9d","Type":"ContainerStarted","Data":"b7e74ad576317dce06d61fb6b148823f5c69cd10ead4f7ca003df60b6c73a708"}
Jul 09 18:05:47.054705 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:47.054696    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-846sw" event={"ID":"80b70a46-af4d-46ce-9ef4-443bac043d9d","Type":"ContainerStarted","Data":"82fae48f296052a30c3f9b38156d9d8cc2805e5591150b74f0885dcf433f2b98"}
Jul 09 18:05:47.077403 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:47.077357    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-846sw" podStartSLOduration=130.343420486 podStartE2EDuration="2m12.077343659s" totalImagesPullingTime="1.733923173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:47.075934512 +0000 UTC m=+133.046662411" watchObservedRunningTime="2026-07-09 18:05:47.077343659 +0000 UTC m=+133.048071558"
Jul 09 18:05:47.239970 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:47.239929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.358122 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:47.358023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:47.899508 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:47.899478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:48.037984 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:48.037954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:49.501396 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:49.501362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:50.474551 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:50.474516    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66c5b75bfb-27257"]
Jul 09 18:05:50.602505 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:50.602467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:50.704067 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:50.704034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:50.802628 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:50.802542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:51.718489 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:51.718455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:51.925390 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:51.925353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:52.029902 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:52.029818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:53.865711 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:53.865657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:54.383144 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:54.383088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:54.466913 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:54.466873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:54.660271 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:54.660239    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:54.865267 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:54.865233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:55.032281 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:55.032170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.317235 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:56.317202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.423042 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:56.423009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.514084 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:56.514051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.624407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:56.624324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:56.698263 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:56.698238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.080505 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:57.080471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:05:57.188297 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:05:57.188267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:02.215731 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:02.215699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:02.451068 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:02.451037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:02.573410 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:02.573322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:08.470564 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:08.470522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:11.614666 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:11.614601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:13.233842 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:13.233794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:13.318143 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:13.318085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:14.469406 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:14.469364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:14.547812 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:14.547776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:15.495028 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.494986    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" podUID="ff49e4bf-776a-4831-a62a-1e1322b98355" containerName="registry" containerID="cri-o://68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8" gracePeriod=30
Jul 09 18:06:15.732768 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.732745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:06:15.765607 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765542    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-installation-pull-secrets\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.765607 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765581    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-bound-sa-token\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.765798 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765610    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-certificates\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.765798 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765782    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9644\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-kube-api-access-h9644\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.765906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-image-registry-private-configuration\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.765906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-trusted-ca\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.766002 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.766002 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.765956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff49e4bf-776a-4831-a62a-1e1322b98355-ca-trust-extracted\") pod \"ff49e4bf-776a-4831-a62a-1e1322b98355\" (UID: \"ff49e4bf-776a-4831-a62a-1e1322b98355\") "
Jul 09 18:06:15.766153 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.766031    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:15.766370 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.766342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:06:15.766455 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.766349    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-certificates\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.768683 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.768640    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-kube-api-access-h9644" (OuterVolumeSpecName: "kube-api-access-h9644") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "kube-api-access-h9644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:15.768789 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.768771    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:15.768883 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.768839    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:06:15.768995 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.768949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:15.768995 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.768942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:06:15.774820 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.774797    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff49e4bf-776a-4831-a62a-1e1322b98355-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ff49e4bf-776a-4831-a62a-1e1322b98355" (UID: "ff49e4bf-776a-4831-a62a-1e1322b98355"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:06:15.867102 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867073    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-image-registry-private-configuration\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.867230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867123    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff49e4bf-776a-4831-a62a-1e1322b98355-trusted-ca\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.867230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867132    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-registry-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.867230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867143    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff49e4bf-776a-4831-a62a-1e1322b98355-ca-trust-extracted\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.867230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867151    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff49e4bf-776a-4831-a62a-1e1322b98355-installation-pull-secrets\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.867230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867159    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-bound-sa-token\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:15.867230 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:15.867167    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9644\" (UniqueName: \"kubernetes.io/projected/ff49e4bf-776a-4831-a62a-1e1322b98355-kube-api-access-h9644\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:06:16.147404 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.147375    2562 generic.go:356] "Generic (PLEG): container finished" podID="ff49e4bf-776a-4831-a62a-1e1322b98355" containerID="68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8" exitCode=0
Jul 09 18:06:16.147561 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.147418    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66c5b75bfb-27257"
Jul 09 18:06:16.147561 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.147448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" event={"ID":"ff49e4bf-776a-4831-a62a-1e1322b98355","Type":"ContainerDied","Data":"68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8"}
Jul 09 18:06:16.147561 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.147487    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66c5b75bfb-27257" event={"ID":"ff49e4bf-776a-4831-a62a-1e1322b98355","Type":"ContainerDied","Data":"a46082cd305c754453f9d4f7411fdf4973f0a09021c5f0173245cf2f351f736e"}
Jul 09 18:06:16.147561 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.147520    2562 scope.go:117] "RemoveContainer" containerID="68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8"
Jul 09 18:06:16.155865 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.155844    2562 scope.go:117] "RemoveContainer" containerID="68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8"
Jul 09 18:06:16.156135 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:06:16.156117    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8\": container with ID starting with 68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8 not found: ID does not exist" containerID="68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8"
Jul 09 18:06:16.156193 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.156143    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8"} err="failed to get container status \"68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8\": rpc error: code = NotFound desc = could not find container \"68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8\": container with ID starting with 68ee070bd94f672585e99be5284d3b03351c97bbc2311df5491fb8acc9a254b8 not found: ID does not exist"
Jul 09 18:06:16.173245 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.170258    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66c5b75bfb-27257"]
Jul 09 18:06:16.174994 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.174974    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66c5b75bfb-27257"]
Jul 09 18:06:16.460665 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.460574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:16.536000 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.535959    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:16.562562 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:16.562531    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff49e4bf-776a-4831-a62a-1e1322b98355" path="/var/lib/kubelet/pods/ff49e4bf-776a-4831-a62a-1e1322b98355/volumes"
Jul 09 18:06:18.300579 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:18.300539    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:18.400791 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:18.400752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:21.286880 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:21.286835    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" podUID="6a5a3065-70ae-4c34-9e54-f65bb7c7f263" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:06:26.624020 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:26.623934    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:26.751592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:26.751556    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:31.286284 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:31.286247    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" podUID="6a5a3065-70ae-4c34-9e54-f65bb7c7f263" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:06:41.286895 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:41.286851    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" podUID="6a5a3065-70ae-4c34-9e54-f65bb7c7f263" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 18:06:41.287356 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:41.286927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v"
Jul 09 18:06:41.287431 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:41.287412    2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f1aee1f9e74cbe81c38df5fa364211e6e8c370c039ebbfb4ae8e461955168a6c"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 09 18:06:41.287472 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:41.287452    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" podUID="6a5a3065-70ae-4c34-9e54-f65bb7c7f263" containerName="service-proxy" containerID="cri-o://f1aee1f9e74cbe81c38df5fa364211e6e8c370c039ebbfb4ae8e461955168a6c" gracePeriod=30
Jul 09 18:06:42.231848 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:42.231808    2562 generic.go:356] "Generic (PLEG): container finished" podID="6a5a3065-70ae-4c34-9e54-f65bb7c7f263" containerID="f1aee1f9e74cbe81c38df5fa364211e6e8c370c039ebbfb4ae8e461955168a6c" exitCode=2
Jul 09 18:06:42.232036 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:42.231892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" event={"ID":"6a5a3065-70ae-4c34-9e54-f65bb7c7f263","Type":"ContainerDied","Data":"f1aee1f9e74cbe81c38df5fa364211e6e8c370c039ebbfb4ae8e461955168a6c"}
Jul 09 18:06:42.232036 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:42.231941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8fb6744bb-jdl9v" event={"ID":"6a5a3065-70ae-4c34-9e54-f65bb7c7f263","Type":"ContainerStarted","Data":"bb0726a46e6d7bab421a05add1b191a2dcad3e721ee57faf534dc82cab415c86"}
Jul 09 18:06:59.328058 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:59.328028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:06:59.429264 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:06:59.429224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:11.269507 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:11.269466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:11.349833 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:11.349798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:26.595659 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:26.595603    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:26.695967 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:26.695934    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:29.524448 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:29.524405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:29.588245 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:29.588199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:29.663539 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:29.663502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:29.764630 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:29.764594    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:30.266545 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:30.266518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:30.339606 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:30.339570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:40.055925 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:40.055884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:40.158387 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:40.158339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:40.258324 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:40.258294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:42.297921 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:42.297888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:51.771897 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:51.771858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:07:51.872025 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:07:51.871981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:01.291945 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:01.291905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:01.380394 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:01.380358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:07.978667 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:07.978623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:29.101161 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:29.101121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:29.196830 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:29.196795    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:29.458668 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:29.458636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:29.557504 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:29.557469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:29.658395 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:29.658358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:34.456717 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:34.456688    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:08:34.460585 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:34.460563    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:08:34.461367 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:34.461349    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:08:37.024762 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:37.024724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:53.800714 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:53.800672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:53.880326 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:53.880286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:57.895611 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:57.895564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:58.310360 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:58.310272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:58.403483 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:58.403438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.105396 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.105349    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.534939 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.534903    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.636083 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.636053    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-9l9x6"]
Jul 09 18:08:59.636371 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.636356    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff49e4bf-776a-4831-a62a-1e1322b98355" containerName="registry"
Jul 09 18:08:59.636371 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.636373    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff49e4bf-776a-4831-a62a-1e1322b98355" containerName="registry"
Jul 09 18:08:59.636454 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.636412    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff49e4bf-776a-4831-a62a-1e1322b98355" containerName="registry"
Jul 09 18:08:59.639194 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.639166    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.659249 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.659226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 09 18:08:59.665999 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.665982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 09 18:08:59.666087 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.666004    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 18:08:59.666216 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.666205    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 18:08:59.675914 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.675900    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-74dfh\""
Jul 09 18:08:59.705307 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.705284    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-9l9x6"]
Jul 09 18:08:59.708064 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.708038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.764963 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.764938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzqtf\" (UniqueName: \"kubernetes.io/projected/e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc-kube-api-access-rzqtf\") pod \"keda-admission-5fbd5c4644-9l9x6\" (UID: \"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc\") " pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.765060 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.764972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc-certificates\") pod \"keda-admission-5fbd5c4644-9l9x6\" (UID: \"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc\") " pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.825026 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.824960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:08:59.866210 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.866185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzqtf\" (UniqueName: \"kubernetes.io/projected/e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc-kube-api-access-rzqtf\") pod \"keda-admission-5fbd5c4644-9l9x6\" (UID: \"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc\") " pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.866310 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.866226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc-certificates\") pod \"keda-admission-5fbd5c4644-9l9x6\" (UID: \"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc\") " pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.868732 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.868701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc-certificates\") pod \"keda-admission-5fbd5c4644-9l9x6\" (UID: \"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc\") " pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.907941 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.907913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzqtf\" (UniqueName: \"kubernetes.io/projected/e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc-kube-api-access-rzqtf\") pod \"keda-admission-5fbd5c4644-9l9x6\" (UID: \"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc\") " pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.948926 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.948905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:08:59.962815 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:08:59.962789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:00.085227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:00.085146    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-9l9x6"]
Jul 09 18:09:00.089268 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:09:00.089246    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f4d333_6aa4_47fa_8be5_a0f28de5a2fc.slice/crio-1a212ef971440febe0f7caedd31aa990d159bb5c245fd4bddce9c542af67568b WatchSource:0}: Error finding container 1a212ef971440febe0f7caedd31aa990d159bb5c245fd4bddce9c542af67568b: Status 404 returned error can't find the container with id 1a212ef971440febe0f7caedd31aa990d159bb5c245fd4bddce9c542af67568b
Jul 09 18:09:00.091012 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:00.090993    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:09:00.305016 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:00.304984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:00.657455 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:00.657422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:00.660160 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:00.660125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6" event={"ID":"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc","Type":"ContainerStarted","Data":"1a212ef971440febe0f7caedd31aa990d159bb5c245fd4bddce9c542af67568b"}
Jul 09 18:09:00.943595 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:00.943491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:02.614049 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:02.613997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:04.692965 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:04.692926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:04.764221 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:04.764189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:05.679060 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:05.679024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6" event={"ID":"e8f4d333-6aa4-47fa-8be5-a0f28de5a2fc","Type":"ContainerStarted","Data":"e1440c57356fa06be68354397aa8db48f6788e80c93d44c5341baa601da50401"}
Jul 09 18:09:05.679247 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:05.679127    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:09:05.705352 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:05.705307    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6" podStartSLOduration=2.001797611 podStartE2EDuration="6.70529187s" totalImagesPullingTime="4.703494259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:05.702904095 +0000 UTC m=+331.673631994" watchObservedRunningTime="2026-07-09 18:09:05.70529187 +0000 UTC m=+331.676019768"
Jul 09 18:09:26.687716 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:26.687635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-9l9x6"
Jul 09 18:09:39.927807 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:39.927767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:09:41.241298 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:09:41.241261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:08.046059 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.046021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:08.143052 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.143017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:08.293608 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.293569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:08.332515 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.332426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-vpxrz"]
Jul 09 18:10:08.335755 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.335732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.346687 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:10:08.346640    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"mlpipeline-s3-artifact\" is forbidden: User \"system:node:ip-10-0-142-14.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-142-14.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" type="*v1.Secret"
Jul 09 18:10:08.346807 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.346723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 18:10:08.347946 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.347930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8qfgx\""
Jul 09 18:10:08.348028 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.347930    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 18:10:08.419549 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.419512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8xf\" (UniqueName: \"kubernetes.io/projected/ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce-kube-api-access-qw8xf\") pod \"seaweedfs-64568bcd49-vpxrz\" (UID: \"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce\") " pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.419717 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.419563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce-data\") pod \"seaweedfs-64568bcd49-vpxrz\" (UID: \"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce\") " pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.434252 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.434218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:08.438671 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.438640    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vpxrz"]
Jul 09 18:10:08.520532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.520484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8xf\" (UniqueName: \"kubernetes.io/projected/ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce-kube-api-access-qw8xf\") pod \"seaweedfs-64568bcd49-vpxrz\" (UID: \"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce\") " pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.520532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.520539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce-data\") pod \"seaweedfs-64568bcd49-vpxrz\" (UID: \"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce\") " pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.520879 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.520864    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce-data\") pod \"seaweedfs-64568bcd49-vpxrz\" (UID: \"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce\") " pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.534278 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.534249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8xf\" (UniqueName: \"kubernetes.io/projected/ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce-kube-api-access-qw8xf\") pod \"seaweedfs-64568bcd49-vpxrz\" (UID: \"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce\") " pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.644897 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.644869    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:08.790372 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.790339    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vpxrz"]
Jul 09 18:10:08.793654 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:10:08.793616    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5af5b1_86a3_43c8_a2c5_c4e4a797ddce.slice/crio-7c49e13a24072334138aafe36a0cb77e0296454486d0a43e1845c4d3d871ffe2 WatchSource:0}: Error finding container 7c49e13a24072334138aafe36a0cb77e0296454486d0a43e1845c4d3d871ffe2: Status 404 returned error can't find the container with id 7c49e13a24072334138aafe36a0cb77e0296454486d0a43e1845c4d3d871ffe2
Jul 09 18:10:08.881414 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.881375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vpxrz" event={"ID":"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce","Type":"ContainerStarted","Data":"7c49e13a24072334138aafe36a0cb77e0296454486d0a43e1845c4d3d871ffe2"}
Jul 09 18:10:08.965423 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:08.965337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:09.232244 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:10:09.232153    2562 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/flysangel/chrislusf/seaweedfs@sha256:a27e9c432f1dfaafb5bb3f5b39065c0df7a15423a3894025c714b3f06f998aeb"
Jul 09 18:10:09.232578 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:10:09.232280    2562 kuberuntime_manager.go:1459] "Unhandled Error" err="container seaweedfs start failed in pod seaweedfs-64568bcd49-vpxrz_kserve(ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce): ErrImagePull: unable to pull image or OCI artifact: pull image err: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 09 18:10:09.232578 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:10:09.232320    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"seaweedfs\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kserve/seaweedfs-64568bcd49-vpxrz" podUID="ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce"
Jul 09 18:10:09.873556 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:09.873524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:09.884960 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:10:09.884926    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"seaweedfs\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/flysangel/chrislusf/seaweedfs@sha256:a27e9c432f1dfaafb5bb3f5b39065c0df7a15423a3894025c714b3f06f998aeb\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kserve/seaweedfs-64568bcd49-vpxrz" podUID="ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce"
Jul 09 18:10:09.902654 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:09.902632    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 18:10:09.970536 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:09.970505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:10:26.945475 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:26.945441    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vpxrz" event={"ID":"ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce","Type":"ContainerStarted","Data":"6860af52db4902954223250e48b79d08d8d8fd144e81ba50b052b2e2aed587e8"}
Jul 09 18:10:26.945841 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:26.945639    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:10:26.965200 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:26.965153    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-vpxrz" podStartSLOduration=1.383679001 podStartE2EDuration="18.965140916s" totalImagesPullingTime="17.581461915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 18:10:26.964633068 +0000 UTC m=+412.935360970" watchObservedRunningTime="2026-07-09 18:10:26.965140916 +0000 UTC m=+412.935868818"
Jul 09 18:10:32.950812 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:10:32.950779    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-vpxrz"
Jul 09 18:11:08.658333 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:08.658299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:08.740891 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:08.740864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:20.023472 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.023430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:20.133970 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.133937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:20.224202 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.224157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:20.721530 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.721486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:20.787595 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.787557    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6h5j2"]
Jul 09 18:11:20.792141 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.792125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:20.801738 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.801716    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:11:20.804270 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.804254    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r28ss\""
Jul 09 18:11:20.806487 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.806458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6h5j2"]
Jul 09 18:11:20.841749 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.841717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:20.924913 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.924883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31c22e81-ed30-4bb7-999c-ba5b12eb950f-cert\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:20.925089 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:20.924939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnx8\" (UniqueName: \"kubernetes.io/projected/31c22e81-ed30-4bb7-999c-ba5b12eb950f-kube-api-access-ttnx8\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.025842 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.025759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnx8\" (UniqueName: \"kubernetes.io/projected/31c22e81-ed30-4bb7-999c-ba5b12eb950f-kube-api-access-ttnx8\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.025842 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.025811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31c22e81-ed30-4bb7-999c-ba5b12eb950f-cert\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.026274 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:11:21.025909    2562 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 18:11:21.026274 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:11:21.025961    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31c22e81-ed30-4bb7-999c-ba5b12eb950f-cert podName:31c22e81-ed30-4bb7-999c-ba5b12eb950f nodeName:}" failed. No retries permitted until 2026-07-09 18:11:21.525946064 +0000 UTC m=+467.496673941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/31c22e81-ed30-4bb7-999c-ba5b12eb950f-cert") pod "odh-model-controller-84b54b7f97-6h5j2" (UID: "31c22e81-ed30-4bb7-999c-ba5b12eb950f") : secret "odh-model-controller-webhook-cert" not found
Jul 09 18:11:21.041123 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.041083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnx8\" (UniqueName: \"kubernetes.io/projected/31c22e81-ed30-4bb7-999c-ba5b12eb950f-kube-api-access-ttnx8\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.062495 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.062465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:21.174140 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.174084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:21.529016 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.528963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31c22e81-ed30-4bb7-999c-ba5b12eb950f-cert\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.531456 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.531435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31c22e81-ed30-4bb7-999c-ba5b12eb950f-cert\") pod \"odh-model-controller-84b54b7f97-6h5j2\" (UID: \"31c22e81-ed30-4bb7-999c-ba5b12eb950f\") " pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.701592 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.701556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:21.775148 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.775060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:21.834144 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.834122    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6h5j2"]
Jul 09 18:11:21.836804 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:11:21.836770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c22e81_ed30_4bb7_999c_ba5b12eb950f.slice/crio-e14203fd7c89361dc2537f5d5c40ac4f57f04a5a0e64e9a84c7b34518e98c7fe WatchSource:0}: Error finding container e14203fd7c89361dc2537f5d5c40ac4f57f04a5a0e64e9a84c7b34518e98c7fe: Status 404 returned error can't find the container with id e14203fd7c89361dc2537f5d5c40ac4f57f04a5a0e64e9a84c7b34518e98c7fe
Jul 09 18:11:21.872963 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:21.872930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:11:22.129083 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:22.128993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-6h5j2" event={"ID":"31c22e81-ed30-4bb7-999c-ba5b12eb950f","Type":"ContainerStarted","Data":"e14203fd7c89361dc2537f5d5c40ac4f57f04a5a0e64e9a84c7b34518e98c7fe"}
Jul 09 18:11:26.145863 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:26.145824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-6h5j2" event={"ID":"31c22e81-ed30-4bb7-999c-ba5b12eb950f","Type":"ContainerStarted","Data":"7f2f56e5d557ea0fe3d13fcf88ed471a3ae3b0d4e3fc4c124959620cdfd07b8e"}
Jul 09 18:11:26.146320 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:26.145900    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:26.165554 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:26.165504    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-6h5j2" podStartSLOduration=2.51700349 podStartE2EDuration="6.165489041s" totalImagesPullingTime="3.648485551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:26.164697299 +0000 UTC m=+472.135425198" watchObservedRunningTime="2026-07-09 18:11:26.165489041 +0000 UTC m=+472.136216940"
Jul 09 18:11:37.151605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:37.151574    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-6h5j2"
Jul 09 18:11:38.034429 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.034394    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gtczz"]
Jul 09 18:11:38.037516 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.037499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.044777 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.044756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73420cbd-4486-4a2a-afaf-abe57464ebf5-model-cache\") pod \"s3-init-gtczz\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") " pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.044877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.044815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpncj\" (UniqueName: \"kubernetes.io/projected/73420cbd-4486-4a2a-afaf-abe57464ebf5-kube-api-access-cpncj\") pod \"s3-init-gtczz\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") " pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.048682 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.048658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gtczz"]
Jul 09 18:11:38.145537 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.145503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpncj\" (UniqueName: \"kubernetes.io/projected/73420cbd-4486-4a2a-afaf-abe57464ebf5-kube-api-access-cpncj\") pod \"s3-init-gtczz\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") " pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.145708 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.145560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73420cbd-4486-4a2a-afaf-abe57464ebf5-model-cache\") pod \"s3-init-gtczz\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") " pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.145879 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.145865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73420cbd-4486-4a2a-afaf-abe57464ebf5-model-cache\") pod \"s3-init-gtczz\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") " pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.155629 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.155605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpncj\" (UniqueName: \"kubernetes.io/projected/73420cbd-4486-4a2a-afaf-abe57464ebf5-kube-api-access-cpncj\") pod \"s3-init-gtczz\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") " pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.346593 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.346519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gtczz"
Jul 09 18:11:38.471957 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:38.471922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gtczz"]
Jul 09 18:11:38.474609 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:11:38.474573    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73420cbd_4486_4a2a_afaf_abe57464ebf5.slice/crio-58c1751a75d3f8dffa23582ae9b4ff3d48ed782b7f78dd3d8cf865d76662d564 WatchSource:0}: Error finding container 58c1751a75d3f8dffa23582ae9b4ff3d48ed782b7f78dd3d8cf865d76662d564: Status 404 returned error can't find the container with id 58c1751a75d3f8dffa23582ae9b4ff3d48ed782b7f78dd3d8cf865d76662d564
Jul 09 18:11:39.191749 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:39.191698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gtczz" event={"ID":"73420cbd-4486-4a2a-afaf-abe57464ebf5","Type":"ContainerStarted","Data":"58c1751a75d3f8dffa23582ae9b4ff3d48ed782b7f78dd3d8cf865d76662d564"}
Jul 09 18:11:43.210456 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:43.210364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gtczz" event={"ID":"73420cbd-4486-4a2a-afaf-abe57464ebf5","Type":"ContainerStarted","Data":"f2298cfe61e117b880da94d21905ef4292a1444ae6de74bee925acd757334546"}
Jul 09 18:11:45.218809 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:45.218773    2562 generic.go:356] "Generic (PLEG): container finished" podID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerID="f2298cfe61e117b880da94d21905ef4292a1444ae6de74bee925acd757334546" exitCode=0
Jul 09 18:11:45.219208 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:45.218845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gtczz" event={"ID":"73420cbd-4486-4a2a-afaf-abe57464ebf5","Type":"ContainerDied","Data":"f2298cfe61e117b880da94d21905ef4292a1444ae6de74bee925acd757334546"}
Jul 09 18:11:50.240355 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:50.240264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gtczz" event={"ID":"73420cbd-4486-4a2a-afaf-abe57464ebf5","Type":"ContainerStarted","Data":"e109dc7a8d391cc51e3649f94ee74bdf7c48f7f89513da15584809667fb13cfe"}
Jul 09 18:11:50.262216 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:11:50.262140    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-gtczz" podStartSLOduration=1.633029768 podStartE2EDuration="12.262117114s" totalImagesPullingTime="9.142949904s" totalInitContainerRuntime="1.486137442s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:50.261867057 +0000 UTC m=+496.232594957" watchObservedRunningTime="2026-07-09 18:11:50.262117114 +0000 UTC m=+496.232845009"
Jul 09 18:12:55.458618 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:55.458580    2562 generic.go:356] "Generic (PLEG): container finished" podID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerID="e109dc7a8d391cc51e3649f94ee74bdf7c48f7f89513da15584809667fb13cfe" exitCode=0
Jul 09 18:12:55.459077 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:55.458626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gtczz" event={"ID":"73420cbd-4486-4a2a-afaf-abe57464ebf5","Type":"ContainerDied","Data":"e109dc7a8d391cc51e3649f94ee74bdf7c48f7f89513da15584809667fb13cfe"}
Jul 09 18:12:56.580814 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.580790    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gtczz"
Jul 09 18:12:56.722050 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.721969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73420cbd-4486-4a2a-afaf-abe57464ebf5-model-cache\") pod \"73420cbd-4486-4a2a-afaf-abe57464ebf5\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") "
Jul 09 18:12:56.722210 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.722055    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpncj\" (UniqueName: \"kubernetes.io/projected/73420cbd-4486-4a2a-afaf-abe57464ebf5-kube-api-access-cpncj\") pod \"73420cbd-4486-4a2a-afaf-abe57464ebf5\" (UID: \"73420cbd-4486-4a2a-afaf-abe57464ebf5\") "
Jul 09 18:12:56.724391 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.724366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73420cbd-4486-4a2a-afaf-abe57464ebf5-kube-api-access-cpncj" (OuterVolumeSpecName: "kube-api-access-cpncj") pod "73420cbd-4486-4a2a-afaf-abe57464ebf5" (UID: "73420cbd-4486-4a2a-afaf-abe57464ebf5"). InnerVolumeSpecName "kube-api-access-cpncj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:56.740661 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.740637    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73420cbd-4486-4a2a-afaf-abe57464ebf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "73420cbd-4486-4a2a-afaf-abe57464ebf5" (UID: "73420cbd-4486-4a2a-afaf-abe57464ebf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:56.822513 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.822487    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73420cbd-4486-4a2a-afaf-abe57464ebf5-model-cache\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:12:56.822513 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:56.822510    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpncj\" (UniqueName: \"kubernetes.io/projected/73420cbd-4486-4a2a-afaf-abe57464ebf5-kube-api-access-cpncj\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:12:57.467246 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:57.467216    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gtczz"
Jul 09 18:12:57.467404 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:57.467225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gtczz" event={"ID":"73420cbd-4486-4a2a-afaf-abe57464ebf5","Type":"ContainerDied","Data":"58c1751a75d3f8dffa23582ae9b4ff3d48ed782b7f78dd3d8cf865d76662d564"}
Jul 09 18:12:57.467404 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:57.467326    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c1751a75d3f8dffa23582ae9b4ff3d48ed782b7f78dd3d8cf865d76662d564"
Jul 09 18:12:58.256666 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:12:58.256637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:07.601498 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.601466    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-rq9rh"]
Jul 09 18:13:07.601872 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.601727    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerName="download-hf-model"
Jul 09 18:13:07.601872 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.601738    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerName="download-hf-model"
Jul 09 18:13:07.601872 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.601748    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerName="s3-init"
Jul 09 18:13:07.601872 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.601754    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerName="s3-init"
Jul 09 18:13:07.601872 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.601801    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="73420cbd-4486-4a2a-afaf-abe57464ebf5" containerName="s3-init"
Jul 09 18:13:07.604684 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.604669    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.607995 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.607972    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 09 18:13:07.616231 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.616208    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rq9rh"]
Jul 09 18:13:07.692992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.692957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-model-cache\") pod \"s3-tls-init-custom-rq9rh\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") " pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.692992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.692998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88ln\" (UniqueName: \"kubernetes.io/projected/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-kube-api-access-h88ln\") pod \"s3-tls-init-custom-rq9rh\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") " pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.793409 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.793363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-model-cache\") pod \"s3-tls-init-custom-rq9rh\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") " pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.793409 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.793414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88ln\" (UniqueName: \"kubernetes.io/projected/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-kube-api-access-h88ln\") pod \"s3-tls-init-custom-rq9rh\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") " pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.793810 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.793788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-model-cache\") pod \"s3-tls-init-custom-rq9rh\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") " pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.803944 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.803911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88ln\" (UniqueName: \"kubernetes.io/projected/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-kube-api-access-h88ln\") pod \"s3-tls-init-custom-rq9rh\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") " pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:07.913343 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:07.913307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:08.049571 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:08.049539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rq9rh"]
Jul 09 18:13:08.053690 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:13:08.053665    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97871dfd_e6ef_4bc0_8402_e21ad0c5feb5.slice/crio-8ad4505e14af022c3fd035fc3bd77de389f2cb2a0ac46719709346f178d92c1e WatchSource:0}: Error finding container 8ad4505e14af022c3fd035fc3bd77de389f2cb2a0ac46719709346f178d92c1e: Status 404 returned error can't find the container with id 8ad4505e14af022c3fd035fc3bd77de389f2cb2a0ac46719709346f178d92c1e
Jul 09 18:13:08.507071 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:08.507031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rq9rh" event={"ID":"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5","Type":"ContainerStarted","Data":"aa82101d1e428e88d2d26f5f386640f63abfabd0785c49f26d1876bb0e8d52ab"}
Jul 09 18:13:08.507292 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:08.507080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rq9rh" event={"ID":"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5","Type":"ContainerStarted","Data":"8ad4505e14af022c3fd035fc3bd77de389f2cb2a0ac46719709346f178d92c1e"}
Jul 09 18:13:09.515464 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:09.515428    2562 generic.go:356] "Generic (PLEG): container finished" podID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerID="aa82101d1e428e88d2d26f5f386640f63abfabd0785c49f26d1876bb0e8d52ab" exitCode=0
Jul 09 18:13:09.515859 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:09.515492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rq9rh" event={"ID":"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5","Type":"ContainerDied","Data":"aa82101d1e428e88d2d26f5f386640f63abfabd0785c49f26d1876bb0e8d52ab"}
Jul 09 18:13:10.520284 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:10.520247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rq9rh" event={"ID":"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5","Type":"ContainerStarted","Data":"b95a29ae5a335a831c2406e0bfe3e85d092ed80f0ed6fde7f89d4081b8deafed"}
Jul 09 18:13:10.538891 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:10.538838    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-rq9rh" podStartSLOduration=2.32029832 podStartE2EDuration="3.538821843s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.218523523s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:10.53846719 +0000 UTC m=+576.509195090" watchObservedRunningTime="2026-07-09 18:13:10.538821843 +0000 UTC m=+576.509549774"
Jul 09 18:13:16.542057 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:16.542021    2562 generic.go:356] "Generic (PLEG): container finished" podID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerID="b95a29ae5a335a831c2406e0bfe3e85d092ed80f0ed6fde7f89d4081b8deafed" exitCode=0
Jul 09 18:13:16.542431 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:16.542115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rq9rh" event={"ID":"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5","Type":"ContainerDied","Data":"b95a29ae5a335a831c2406e0bfe3e85d092ed80f0ed6fde7f89d4081b8deafed"}
Jul 09 18:13:17.679779 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.679756    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:17.770409 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.770375    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-model-cache\") pod \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") "
Jul 09 18:13:17.770568 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.770429    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h88ln\" (UniqueName: \"kubernetes.io/projected/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-kube-api-access-h88ln\") pod \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\" (UID: \"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5\") "
Jul 09 18:13:17.772622 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.772596    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-kube-api-access-h88ln" (OuterVolumeSpecName: "kube-api-access-h88ln") pod "97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" (UID: "97871dfd-e6ef-4bc0-8402-e21ad0c5feb5"). InnerVolumeSpecName "kube-api-access-h88ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:13:17.791933 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.791908    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" (UID: "97871dfd-e6ef-4bc0-8402-e21ad0c5feb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:13:17.871857 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.871786    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88ln\" (UniqueName: \"kubernetes.io/projected/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-kube-api-access-h88ln\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:13:17.871857 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:17.871814    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97871dfd-e6ef-4bc0-8402-e21ad0c5feb5-model-cache\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:13:18.555992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:18.555958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rq9rh" event={"ID":"97871dfd-e6ef-4bc0-8402-e21ad0c5feb5","Type":"ContainerDied","Data":"8ad4505e14af022c3fd035fc3bd77de389f2cb2a0ac46719709346f178d92c1e"}
Jul 09 18:13:18.555992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:18.555998    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad4505e14af022c3fd035fc3bd77de389f2cb2a0ac46719709346f178d92c1e"
Jul 09 18:13:18.556289 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:18.556012    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rq9rh"
Jul 09 18:13:19.176821 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:19.176786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:19.709608 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:19.709576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:21.985955 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.985916    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-jp4ph"]
Jul 09 18:13:21.986350 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.986188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerName="download-hf-model"
Jul 09 18:13:21.986350 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.986201    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerName="download-hf-model"
Jul 09 18:13:21.986350 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.986209    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerName="s3-tls-init-custom"
Jul 09 18:13:21.986350 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.986215    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerName="s3-tls-init-custom"
Jul 09 18:13:21.986350 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.986264    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="97871dfd-e6ef-4bc0-8402-e21ad0c5feb5" containerName="s3-tls-init-custom"
Jul 09 18:13:21.991125 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.991082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:21.995752 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:21.995737    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 09 18:13:22.008106 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.008062    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-jp4ph"]
Jul 09 18:13:22.100632 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.100598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24rz\" (UniqueName: \"kubernetes.io/projected/be0c1b71-7af6-498f-af34-1da8e40fecf7-kube-api-access-c24rz\") pod \"s3-tls-init-serving-jp4ph\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") " pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.100787 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.100637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c1b71-7af6-498f-af34-1da8e40fecf7-model-cache\") pod \"s3-tls-init-serving-jp4ph\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") " pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.201868 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.201835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24rz\" (UniqueName: \"kubernetes.io/projected/be0c1b71-7af6-498f-af34-1da8e40fecf7-kube-api-access-c24rz\") pod \"s3-tls-init-serving-jp4ph\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") " pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.201993 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.201871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c1b71-7af6-498f-af34-1da8e40fecf7-model-cache\") pod \"s3-tls-init-serving-jp4ph\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") " pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.202235 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.202219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c1b71-7af6-498f-af34-1da8e40fecf7-model-cache\") pod \"s3-tls-init-serving-jp4ph\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") " pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.212569 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.212542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24rz\" (UniqueName: \"kubernetes.io/projected/be0c1b71-7af6-498f-af34-1da8e40fecf7-kube-api-access-c24rz\") pod \"s3-tls-init-serving-jp4ph\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") " pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.300223 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.300142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:22.433690 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.433508    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-jp4ph"]
Jul 09 18:13:22.436451 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:13:22.436423    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe0c1b71_7af6_498f_af34_1da8e40fecf7.slice/crio-347f71e7b1f347e7c9c0b2a2978167f0b123722e9cc1cd7073d6803aaa949f6e WatchSource:0}: Error finding container 347f71e7b1f347e7c9c0b2a2978167f0b123722e9cc1cd7073d6803aaa949f6e: Status 404 returned error can't find the container with id 347f71e7b1f347e7c9c0b2a2978167f0b123722e9cc1cd7073d6803aaa949f6e
Jul 09 18:13:22.570878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.570791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jp4ph" event={"ID":"be0c1b71-7af6-498f-af34-1da8e40fecf7","Type":"ContainerStarted","Data":"6fc09ec6234cd1636f94497bf02b71e1b600b49afd17f4ed5a2706a2ec789e42"}
Jul 09 18:13:22.570878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:22.570831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jp4ph" event={"ID":"be0c1b71-7af6-498f-af34-1da8e40fecf7","Type":"ContainerStarted","Data":"347f71e7b1f347e7c9c0b2a2978167f0b123722e9cc1cd7073d6803aaa949f6e"}
Jul 09 18:13:24.579497 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:24.579463    2562 generic.go:356] "Generic (PLEG): container finished" podID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerID="6fc09ec6234cd1636f94497bf02b71e1b600b49afd17f4ed5a2706a2ec789e42" exitCode=0
Jul 09 18:13:24.579873 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:24.579543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jp4ph" event={"ID":"be0c1b71-7af6-498f-af34-1da8e40fecf7","Type":"ContainerDied","Data":"6fc09ec6234cd1636f94497bf02b71e1b600b49afd17f4ed5a2706a2ec789e42"}
Jul 09 18:13:25.588153 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:25.588085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jp4ph" event={"ID":"be0c1b71-7af6-498f-af34-1da8e40fecf7","Type":"ContainerStarted","Data":"7a4d0a11da112c8b2cf433e208ef4cfbcd5b37ebb35588c0fd793eb978bac04a"}
Jul 09 18:13:25.609001 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:25.608954    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-jp4ph" podStartSLOduration=3.143790489 podStartE2EDuration="4.608939419s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.46514893s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:13:25.608016826 +0000 UTC m=+591.578744724" watchObservedRunningTime="2026-07-09 18:13:25.608939419 +0000 UTC m=+591.579667318"
Jul 09 18:13:31.612827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:31.612740    2562 generic.go:356] "Generic (PLEG): container finished" podID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerID="7a4d0a11da112c8b2cf433e208ef4cfbcd5b37ebb35588c0fd793eb978bac04a" exitCode=0
Jul 09 18:13:31.612827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:31.612811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jp4ph" event={"ID":"be0c1b71-7af6-498f-af34-1da8e40fecf7","Type":"ContainerDied","Data":"7a4d0a11da112c8b2cf433e208ef4cfbcd5b37ebb35588c0fd793eb978bac04a"}
Jul 09 18:13:32.755140 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.755117    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:32.882273 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.882179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c24rz\" (UniqueName: \"kubernetes.io/projected/be0c1b71-7af6-498f-af34-1da8e40fecf7-kube-api-access-c24rz\") pod \"be0c1b71-7af6-498f-af34-1da8e40fecf7\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") "
Jul 09 18:13:32.882273 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.882243    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c1b71-7af6-498f-af34-1da8e40fecf7-model-cache\") pod \"be0c1b71-7af6-498f-af34-1da8e40fecf7\" (UID: \"be0c1b71-7af6-498f-af34-1da8e40fecf7\") "
Jul 09 18:13:32.884387 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.884356    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0c1b71-7af6-498f-af34-1da8e40fecf7-kube-api-access-c24rz" (OuterVolumeSpecName: "kube-api-access-c24rz") pod "be0c1b71-7af6-498f-af34-1da8e40fecf7" (UID: "be0c1b71-7af6-498f-af34-1da8e40fecf7"). InnerVolumeSpecName "kube-api-access-c24rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:13:32.907484 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.907448    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0c1b71-7af6-498f-af34-1da8e40fecf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "be0c1b71-7af6-498f-af34-1da8e40fecf7" (UID: "be0c1b71-7af6-498f-af34-1da8e40fecf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:13:32.983388 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.983358    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c1b71-7af6-498f-af34-1da8e40fecf7-model-cache\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:13:32.983388 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:32.983383    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24rz\" (UniqueName: \"kubernetes.io/projected/be0c1b71-7af6-498f-af34-1da8e40fecf7-kube-api-access-c24rz\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:13:33.620603 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:33.620571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-jp4ph" event={"ID":"be0c1b71-7af6-498f-af34-1da8e40fecf7","Type":"ContainerDied","Data":"347f71e7b1f347e7c9c0b2a2978167f0b123722e9cc1cd7073d6803aaa949f6e"}
Jul 09 18:13:33.620603 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:33.620609    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347f71e7b1f347e7c9c0b2a2978167f0b123722e9cc1cd7073d6803aaa949f6e"
Jul 09 18:13:33.620803 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:33.620584    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-jp4ph"
Jul 09 18:13:34.224346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:34.224311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:34.344553 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:34.344515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:34.481391 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:34.481307    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:13:34.487589 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:34.487566    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:13:45.268585 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.268520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:45.421195 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.421165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:13:45.429210 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.429186    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"]
Jul 09 18:13:45.429455 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.429440    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerName="s3-tls-init-serving"
Jul 09 18:13:45.429497 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.429459    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerName="s3-tls-init-serving"
Jul 09 18:13:45.429497 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.429491    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerName="download-hf-model"
Jul 09 18:13:45.429558 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.429500    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerName="download-hf-model"
Jul 09 18:13:45.429594 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.429561    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0c1b71-7af6-498f-af34-1da8e40fecf7" containerName="s3-tls-init-serving"
Jul 09 18:13:45.432616 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.432599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.437440 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.437419    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:13:45.438046 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.438018    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 09 18:13:45.438198 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.438177    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:13:45.438284 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.438182    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 09 18:13:45.438515 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.438491    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:13:45.438515 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.438497    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:13:45.439159 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.438781    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xqd54\""
Jul 09 18:13:45.454393 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.454370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"]
Jul 09 18:13:45.474173 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.474149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bde02d1-3fba-4ea2-99c5-9fae05786b39-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.474261 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.474183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.474261 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.474250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpvrb\" (UniqueName: \"kubernetes.io/projected/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-api-access-xpvrb\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.474330 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.474313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.474362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.474334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.575578 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.575510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.575578 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.575547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.575736 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.575595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bde02d1-3fba-4ea2-99c5-9fae05786b39-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.575736 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.575620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.575736 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.575651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpvrb\" (UniqueName: \"kubernetes.io/projected/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-api-access-xpvrb\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.575957 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.575934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.576292 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.576266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.576395 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.576374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.578183 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.578156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bde02d1-3fba-4ea2-99c5-9fae05786b39-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.589620 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.589598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpvrb\" (UniqueName: \"kubernetes.io/projected/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-api-access-xpvrb\") pod \"isvc-sklearn-batcher-predictor-66856f8b85-9sbsf\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.742712 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.742686    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:13:45.898728 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:45.898698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"]
Jul 09 18:13:45.901735 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:13:45.901702    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bde02d1_3fba_4ea2_99c5_9fae05786b39.slice/crio-1d569711e20dc7553e3c8b4c36c8aadfba1131d3224ce7d9c41a414a02d11a9b WatchSource:0}: Error finding container 1d569711e20dc7553e3c8b4c36c8aadfba1131d3224ce7d9c41a414a02d11a9b: Status 404 returned error can't find the container with id 1d569711e20dc7553e3c8b4c36c8aadfba1131d3224ce7d9c41a414a02d11a9b
Jul 09 18:13:46.668979 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:46.668930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerStarted","Data":"904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4"}
Jul 09 18:13:46.669472 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:46.669006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerStarted","Data":"1d569711e20dc7553e3c8b4c36c8aadfba1131d3224ce7d9c41a414a02d11a9b"}
Jul 09 18:13:50.685202 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:50.685163    2562 generic.go:356] "Generic (PLEG): container finished" podID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerID="904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4" exitCode=0
Jul 09 18:13:50.685686 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:13:50.685247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerDied","Data":"904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4"}
Jul 09 18:14:04.680242 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:04.680222    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:14:04.749544 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:04.749505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerStarted","Data":"a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636"}
Jul 09 18:14:07.762608 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:07.762575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerStarted","Data":"28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c"}
Jul 09 18:14:10.776526 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:10.776434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerStarted","Data":"75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d"}
Jul 09 18:14:10.776919 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:10.776655    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:14:10.776919 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:10.776809    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:14:10.778007 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:10.777979    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:11.780903 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:11.780866    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:14:11.781358 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:11.780947    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:11.782014 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:11.781982    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:11.784949 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:11.784932    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:14:11.829108 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:11.829056    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podStartSLOduration=3.475809996 podStartE2EDuration="26.829040546s" totalImagesPullingTime="19.572207129s" totalInitContainerRuntime="3.781023421s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:13:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:14:10.81172911 +0000 UTC m=+636.782457009" watchObservedRunningTime="2026-07-09 18:14:11.829040546 +0000 UTC m=+637.799768444"
Jul 09 18:14:12.784185 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:12.784145    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:12.784547 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:12.784428    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:13.788403 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:13.788360    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:13.788783 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:13.788705    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:23.788757 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:23.788711    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:23.789259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:23.789235    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:33.788564 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:33.788522    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:33.789152 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:33.789128    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:43.788484 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:43.788435    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:43.789017 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:43.788828    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:14:53.788369 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:53.788321    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:14:53.788802 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:14:53.788781    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:03.788829 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:03.788776    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:15:03.789300 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:03.789203    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:13.788458 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:13.788414    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:15:13.788950 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:13.788916    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:23.789826 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:23.789793    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:15:23.790373 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:23.789912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:15:30.338297 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.337437    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"]
Jul 09 18:15:30.338297 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.337854    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" containerID="cri-o://a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636" gracePeriod=30
Jul 09 18:15:30.338860 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.338325    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" containerID="cri-o://28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c" gracePeriod=30
Jul 09 18:15:30.339834 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.338089    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" containerID="cri-o://75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d" gracePeriod=30
Jul 09 18:15:30.409916 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.409882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:30.456797 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.456771    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"]
Jul 09 18:15:30.459937 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.459920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.462717 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.462700    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 09 18:15:30.463035 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.463019    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 09 18:15:30.474043 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.474016    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"]
Jul 09 18:15:30.548562 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.548533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kfk\" (UniqueName: \"kubernetes.io/projected/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-api-access-h5kfk\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.548702 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.548571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.548702 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.548597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13b836f2-52ec-4c9c-92ae-f03e0317baeb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.548702 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.548670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.548702 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.548699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.649899 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.649870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650065 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.649919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kfk\" (UniqueName: \"kubernetes.io/projected/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-api-access-h5kfk\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650065 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.649952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650065 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.649980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13b836f2-52ec-4c9c-92ae-f03e0317baeb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650065 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.650032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650399 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.650377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650576 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.650557    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.650702 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.650674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.652684 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.652661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13b836f2-52ec-4c9c-92ae-f03e0317baeb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.661172 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.661150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kfk\" (UniqueName: \"kubernetes.io/projected/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-api-access-h5kfk\") pod \"isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.770759 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.770721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:30.907013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:30.906896    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"]
Jul 09 18:15:30.909078 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:15:30.909052    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b836f2_52ec_4c9c_92ae_f03e0317baeb.slice/crio-c2be5c010888bd553f7e1e3ccdeda40d46cd00619dc7f821388dfb5588c0b266 WatchSource:0}: Error finding container c2be5c010888bd553f7e1e3ccdeda40d46cd00619dc7f821388dfb5588c0b266: Status 404 returned error can't find the container with id c2be5c010888bd553f7e1e3ccdeda40d46cd00619dc7f821388dfb5588c0b266
Jul 09 18:15:31.062728 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:31.062695    2562 generic.go:356] "Generic (PLEG): container finished" podID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerID="28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c" exitCode=2
Jul 09 18:15:31.062912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:31.062775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerDied","Data":"28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c"}
Jul 09 18:15:31.064313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:31.064288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerStarted","Data":"25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4"}
Jul 09 18:15:31.064460 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:31.064342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerStarted","Data":"c2be5c010888bd553f7e1e3ccdeda40d46cd00619dc7f821388dfb5588c0b266"}
Jul 09 18:15:31.781707 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:31.781670    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 09 18:15:32.577798 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:32.577767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:15:33.789014 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:33.788969    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:15:33.789474 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:33.789267    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:35.081638 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:35.081603    2562 generic.go:356] "Generic (PLEG): container finished" podID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerID="a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636" exitCode=0
Jul 09 18:15:35.082050 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:35.081681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerDied","Data":"a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636"}
Jul 09 18:15:35.083064 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:35.083042    2562 generic.go:356] "Generic (PLEG): container finished" podID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerID="25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4" exitCode=0
Jul 09 18:15:35.083205 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:35.083079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerDied","Data":"25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4"}
Jul 09 18:15:36.089428 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.089396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerStarted","Data":"361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24"}
Jul 09 18:15:36.089808 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.089467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerStarted","Data":"759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7"}
Jul 09 18:15:36.089808 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.089493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerStarted","Data":"a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb"}
Jul 09 18:15:36.089808 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.089776    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:36.089964 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.089933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:36.091120 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.091079    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:15:36.115463 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.115422    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podStartSLOduration=2.492469755 podStartE2EDuration="6.115409203s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.622939448s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:15:36.114044783 +0000 UTC m=+722.084772695" watchObservedRunningTime="2026-07-09 18:15:36.115409203 +0000 UTC m=+722.086137101"
Jul 09 18:15:36.782153 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:36.782079    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 09 18:15:37.093741 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:37.093646    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:15:37.093741 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:37.093706    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:37.094894 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:37.094864    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:38.097253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:38.097211    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:15:38.097657 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:38.097633    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:41.782006 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:41.781954    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 09 18:15:41.782424 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:41.782120    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:15:43.101910 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:43.101882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:15:43.102464 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:43.102425    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:15:43.102741 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:43.102717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:43.789263 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:43.789217    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:15:43.789514 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:43.789489    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:46.781275 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:46.781230    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 09 18:15:51.782079 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:51.782035    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 09 18:15:53.102989 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:53.102943    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:15:53.103431 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:53.103406    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:53.788453 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:53.788408    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 09 18:15:53.788608 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:53.788549    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:15:53.788781 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:53.788753    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:15:53.788878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:53.788858    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:15:56.781754 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:15:56.781713    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 09 18:16:00.478020 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.477998    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:16:00.570293 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570260    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-cabundle-cert\") pod \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") "
Jul 09 18:16:00.570460 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570341    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bde02d1-3fba-4ea2-99c5-9fae05786b39-proxy-tls\") pod \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") "
Jul 09 18:16:00.570460 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570439    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-rbac-proxy-sar-config\") pod \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") "
Jul 09 18:16:00.570533 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kserve-provision-location\") pod \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") "
Jul 09 18:16:00.570533 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570515    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpvrb\" (UniqueName: \"kubernetes.io/projected/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-api-access-xpvrb\") pod \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\" (UID: \"9bde02d1-3fba-4ea2-99c5-9fae05786b39\") "
Jul 09 18:16:00.570669 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570638    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9bde02d1-3fba-4ea2-99c5-9fae05786b39" (UID: "9bde02d1-3fba-4ea2-99c5-9fae05786b39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:00.570967 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570844    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9bde02d1-3fba-4ea2-99c5-9fae05786b39" (UID: "9bde02d1-3fba-4ea2-99c5-9fae05786b39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:16:00.570967 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.570858    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9bde02d1-3fba-4ea2-99c5-9fae05786b39" (UID: "9bde02d1-3fba-4ea2-99c5-9fae05786b39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:16:00.572705 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.572684    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bde02d1-3fba-4ea2-99c5-9fae05786b39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bde02d1-3fba-4ea2-99c5-9fae05786b39" (UID: "9bde02d1-3fba-4ea2-99c5-9fae05786b39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:16:00.572826 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.572807    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-api-access-xpvrb" (OuterVolumeSpecName: "kube-api-access-xpvrb") pod "9bde02d1-3fba-4ea2-99c5-9fae05786b39" (UID: "9bde02d1-3fba-4ea2-99c5-9fae05786b39"). InnerVolumeSpecName "kube-api-access-xpvrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:16:00.671675 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.671633    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bde02d1-3fba-4ea2-99c5-9fae05786b39-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:16:00.671675 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.671676    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:16:00.671863 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.671690    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:16:00.671863 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.671703    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpvrb\" (UniqueName: \"kubernetes.io/projected/9bde02d1-3fba-4ea2-99c5-9fae05786b39-kube-api-access-xpvrb\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:16:00.671863 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:00.671716    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9bde02d1-3fba-4ea2-99c5-9fae05786b39-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:16:01.183302 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.183269    2562 generic.go:356] "Generic (PLEG): container finished" podID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerID="75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d" exitCode=0
Jul 09 18:16:01.183436 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.183334    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"
Jul 09 18:16:01.183436 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.183360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerDied","Data":"75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d"}
Jul 09 18:16:01.183436 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.183430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf" event={"ID":"9bde02d1-3fba-4ea2-99c5-9fae05786b39","Type":"ContainerDied","Data":"1d569711e20dc7553e3c8b4c36c8aadfba1131d3224ce7d9c41a414a02d11a9b"}
Jul 09 18:16:01.183552 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.183462    2562 scope.go:117] "RemoveContainer" containerID="75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d"
Jul 09 18:16:01.192130 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.192114    2562 scope.go:117] "RemoveContainer" containerID="28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c"
Jul 09 18:16:01.199180 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.199153    2562 scope.go:117] "RemoveContainer" containerID="a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636"
Jul 09 18:16:01.207125 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.207081    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"]
Jul 09 18:16:01.207308 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.207277    2562 scope.go:117] "RemoveContainer" containerID="904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4"
Jul 09 18:16:01.210313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.210292    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-66856f8b85-9sbsf"]
Jul 09 18:16:01.214199 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.214181    2562 scope.go:117] "RemoveContainer" containerID="75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d"
Jul 09 18:16:01.214476 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:16:01.214457    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d\": container with ID starting with 75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d not found: ID does not exist" containerID="75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d"
Jul 09 18:16:01.214532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.214485    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d"} err="failed to get container status \"75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d\": rpc error: code = NotFound desc = could not find container \"75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d\": container with ID starting with 75c71d865d47e0a7b11c033368d0d3eded575e5abec617af91e2ff66ca97bd0d not found: ID does not exist"
Jul 09 18:16:01.214532 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.214503    2562 scope.go:117] "RemoveContainer" containerID="28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c"
Jul 09 18:16:01.214767 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:16:01.214752    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c\": container with ID starting with 28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c not found: ID does not exist" containerID="28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c"
Jul 09 18:16:01.214810 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.214773    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c"} err="failed to get container status \"28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c\": rpc error: code = NotFound desc = could not find container \"28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c\": container with ID starting with 28699447853cf0ae1548a8a0a9c38c031b61cdaa896b3a9128155447bb6a7f4c not found: ID does not exist"
Jul 09 18:16:01.214810 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.214803    2562 scope.go:117] "RemoveContainer" containerID="a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636"
Jul 09 18:16:01.215041 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:16:01.215025    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636\": container with ID starting with a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636 not found: ID does not exist" containerID="a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636"
Jul 09 18:16:01.215080 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.215047    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636"} err="failed to get container status \"a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636\": rpc error: code = NotFound desc = could not find container \"a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636\": container with ID starting with a10d68ea4b4565cdb09fe5b6b54764a60d74285bd267f8278833317563741636 not found: ID does not exist"
Jul 09 18:16:01.215080 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.215062    2562 scope.go:117] "RemoveContainer" containerID="904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4"
Jul 09 18:16:01.215317 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:16:01.215300    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4\": container with ID starting with 904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4 not found: ID does not exist" containerID="904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4"
Jul 09 18:16:01.215372 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:01.215321    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4"} err="failed to get container status \"904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4\": rpc error: code = NotFound desc = could not find container \"904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4\": container with ID starting with 904f3367e44bec873bd09757775679ed166b578fbcd80836996fa754f53329b4 not found: ID does not exist"
Jul 09 18:16:02.562796 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:02.562763    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" path="/var/lib/kubelet/pods/9bde02d1-3fba-4ea2-99c5-9fae05786b39/volumes"
Jul 09 18:16:03.102804 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:03.102760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:16:03.103248 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:03.103223    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:13.102997 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:13.102946    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:16:13.103424 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:13.103404    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:13.472304 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:13.472271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:13.532605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:13.532576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:16:23.102353 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:23.102308    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:16:23.102848 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:23.102710    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:33.102832 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:33.102788    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:16:33.103251 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:33.103229    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:43.102987 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:43.102937    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:16:43.103488 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:43.103351    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:16:53.103282 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:53.103248    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:16:53.103829 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:16:53.103375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:17:05.583195 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.583148    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"]
Jul 09 18:17:05.583663 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.583607    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" containerID="cri-o://a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb" gracePeriod=30
Jul 09 18:17:05.584131 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.584064    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" containerID="cri-o://361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24" gracePeriod=30
Jul 09 18:17:05.584672 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.584292    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" containerID="cri-o://759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7" gracePeriod=30
Jul 09 18:17:05.612566 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.612541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:05.662960 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.662927    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"]
Jul 09 18:17:05.663206 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663194    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container"
Jul 09 18:17:05.663253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663208    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container"
Jul 09 18:17:05.663253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663217    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy"
Jul 09 18:17:05.663253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663223    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy"
Jul 09 18:17:05.663253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663239    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="storage-initializer"
Jul 09 18:17:05.663253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663245    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="storage-initializer"
Jul 09 18:17:05.663253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663254    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent"
Jul 09 18:17:05.663424 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663260    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent"
Jul 09 18:17:05.663424 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663302    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kserve-container"
Jul 09 18:17:05.663424 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663308    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="agent"
Jul 09 18:17:05.663424 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.663315    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bde02d1-3fba-4ea2-99c5-9fae05786b39" containerName="kube-rbac-proxy"
Jul 09 18:17:05.665877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.665859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.669111 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.669073    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 09 18:17:05.669205 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.669121    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 09 18:17:05.676940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.676919    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"]
Jul 09 18:17:05.699999 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.699957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:05.743398 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.743364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.743398 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.743396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.743570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.743432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mkx\" (UniqueName: \"kubernetes.io/projected/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-api-access-j5mkx\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.843917 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.843819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mkx\" (UniqueName: \"kubernetes.io/projected/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-api-access-j5mkx\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.843917 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.843898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.843917 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.843918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.844245 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:17:05.844054    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 09 18:17:05.844245 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:17:05.844164    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls podName:02a7ed3d-63d6-465f-8165-fe682bedbedd nodeName:}" failed. No retries permitted until 2026-07-09 18:17:06.344141814 +0000 UTC m=+812.314869707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-rcq7l" (UID: "02a7ed3d-63d6-465f-8165-fe682bedbedd") : secret "message-dumper-predictor-serving-cert" not found
Jul 09 18:17:05.844575 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.844555    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.854748 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.854729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mkx\" (UniqueName: \"kubernetes.io/projected/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-api-access-j5mkx\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:05.929052 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:05.929021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:06.347782 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:06.347743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:06.350296 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:06.350271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-rcq7l\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:06.430495 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:06.430463    2562 generic.go:356] "Generic (PLEG): container finished" podID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerID="759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7" exitCode=2
Jul 09 18:17:06.430655 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:06.430529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerDied","Data":"759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7"}
Jul 09 18:17:06.575962 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:06.575928    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:06.707139 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:06.707118    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"]
Jul 09 18:17:06.709442 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:17:06.709411    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a7ed3d_63d6_465f_8165_fe682bedbedd.slice/crio-9c9d11598dfe9f5a2d25e79779b1341eac44f8ae5d2a9d94c089e96d7ef07804 WatchSource:0}: Error finding container 9c9d11598dfe9f5a2d25e79779b1341eac44f8ae5d2a9d94c089e96d7ef07804: Status 404 returned error can't find the container with id 9c9d11598dfe9f5a2d25e79779b1341eac44f8ae5d2a9d94c089e96d7ef07804
Jul 09 18:17:07.436062 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:07.436028    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" event={"ID":"02a7ed3d-63d6-465f-8165-fe682bedbedd","Type":"ContainerStarted","Data":"9c9d11598dfe9f5a2d25e79779b1341eac44f8ae5d2a9d94c089e96d7ef07804"}
Jul 09 18:17:08.097760 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.097730    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 09 18:17:08.442377 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.442333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" event={"ID":"02a7ed3d-63d6-465f-8165-fe682bedbedd","Type":"ContainerStarted","Data":"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"}
Jul 09 18:17:08.442541 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.442412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" event={"ID":"02a7ed3d-63d6-465f-8165-fe682bedbedd","Type":"ContainerStarted","Data":"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"}
Jul 09 18:17:08.442541 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.442450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:08.442541 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.442484    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:08.444180 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.444162    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:08.463034 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:08.462987    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" podStartSLOduration=2.095306899 podStartE2EDuration="3.462973435s" totalImagesPullingTime="1.367666536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:08.461285129 +0000 UTC m=+814.432013029" watchObservedRunningTime="2026-07-09 18:17:08.462973435 +0000 UTC m=+814.433701333"
Jul 09 18:17:10.452645 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:10.452614    2562 generic.go:356] "Generic (PLEG): container finished" podID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerID="a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb" exitCode=0
Jul 09 18:17:10.453009 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:10.452692    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerDied","Data":"a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb"}
Jul 09 18:17:12.308874 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:12.308838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:13.098115 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:13.098061    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 09 18:17:13.102456 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:13.102422    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:17:13.102736 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:13.102710    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:15.458264 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:15.458237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:17:18.097365 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:18.097324    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 09 18:17:18.097803 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:18.097456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:17:18.388483 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:18.388457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:23.097859 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:23.097806    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 09 18:17:23.102375 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:23.102348    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:17:23.102730 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:23.102709    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:25.697381 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.697346    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"]
Jul 09 18:17:25.699547 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.699521    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:25.703481 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.703464    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.706476 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.706439    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 09 18:17:25.706476 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.706472    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 09 18:17:25.713728 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.713702    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"]
Jul 09 18:17:25.792822 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.792793    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:17:25.796915 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.796895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.796979 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.796928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-cabundle-cert\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.796979 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.796955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kserve-provision-location\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.797055 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.797025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnff2\" (UniqueName: \"kubernetes.io/projected/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-api-access-wnff2\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.797111 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.797062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-proxy-tls\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.898376 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.898340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnff2\" (UniqueName: \"kubernetes.io/projected/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-api-access-wnff2\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.898512 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.898389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-proxy-tls\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.898512 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.898431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.898512 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.898451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-cabundle-cert\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.898512 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.898484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kserve-provision-location\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.898991 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.898963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kserve-provision-location\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.899284 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.899263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.899333 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.899263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-cabundle-cert\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.901180 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.901159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-proxy-tls\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:25.911114 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:25.911066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnff2\" (UniqueName: \"kubernetes.io/projected/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-api-access-wnff2\") pod \"isvc-logger-predictor-57ff67ccd6-d7gbk\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:26.014761 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:26.014664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:26.148126 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:26.148084    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"]
Jul 09 18:17:26.150138 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:17:26.150109    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc348671f_a7ef_46ad_9b57_5b5ac2dd5a5c.slice/crio-30c408ca5d217612cee3c86d93f51ce4ba8a698c9d45378960e54e33e8540cfc WatchSource:0}: Error finding container 30c408ca5d217612cee3c86d93f51ce4ba8a698c9d45378960e54e33e8540cfc: Status 404 returned error can't find the container with id 30c408ca5d217612cee3c86d93f51ce4ba8a698c9d45378960e54e33e8540cfc
Jul 09 18:17:26.511643 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:26.511607    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerStarted","Data":"4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc"}
Jul 09 18:17:26.511806 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:26.511683    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerStarted","Data":"30c408ca5d217612cee3c86d93f51ce4ba8a698c9d45378960e54e33e8540cfc"}
Jul 09 18:17:28.098268 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:28.098231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 09 18:17:30.527264 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:30.527227    2562 generic.go:356] "Generic (PLEG): container finished" podID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerID="4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc" exitCode=0
Jul 09 18:17:30.527640 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:30.527298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerDied","Data":"4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc"}
Jul 09 18:17:31.533431 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.533398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerStarted","Data":"22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3"}
Jul 09 18:17:31.533838 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.533478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerStarted","Data":"dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444"}
Jul 09 18:17:31.533838 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.533517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerStarted","Data":"c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c"}
Jul 09 18:17:31.533988 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.533837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:31.533988 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.533898    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:31.533988 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.533932    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:31.535687 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.535651    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:17:31.536362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.536339    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:31.558923 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:31.558877    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podStartSLOduration=2.936114561 podStartE2EDuration="6.558864702s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.622750141s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:31.558047834 +0000 UTC m=+837.528775732" watchObservedRunningTime="2026-07-09 18:17:31.558864702 +0000 UTC m=+837.529592600"
Jul 09 18:17:32.538501 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:32.538455    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:17:32.538962 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:32.538906    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:33.097860 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:33.097819    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 09 18:17:33.103272 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:33.103241    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Jul 09 18:17:33.103407 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:33.103373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:17:33.103584 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:33.103558    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:33.103662 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:33.103649    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:17:35.736895 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.736864    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:17:35.768315 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768285    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13b836f2-52ec-4c9c-92ae-f03e0317baeb-proxy-tls\") pod \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") "
Jul 09 18:17:35.768493 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768325    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5kfk\" (UniqueName: \"kubernetes.io/projected/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-api-access-h5kfk\") pod \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") "
Jul 09 18:17:35.768493 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-rbac-proxy-sar-config\") pod \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") "
Jul 09 18:17:35.768493 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-cabundle-cert\") pod \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") "
Jul 09 18:17:35.768676 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768507    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kserve-provision-location\") pod \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\" (UID: \"13b836f2-52ec-4c9c-92ae-f03e0317baeb\") "
Jul 09 18:17:35.768730 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13b836f2-52ec-4c9c-92ae-f03e0317baeb" (UID: "13b836f2-52ec-4c9c-92ae-f03e0317baeb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:17:35.768783 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13b836f2-52ec-4c9c-92ae-f03e0317baeb" (UID: "13b836f2-52ec-4c9c-92ae-f03e0317baeb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:17:35.768906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.768883    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13b836f2-52ec-4c9c-92ae-f03e0317baeb" (UID: "13b836f2-52ec-4c9c-92ae-f03e0317baeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:35.771070 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.771048    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-api-access-h5kfk" (OuterVolumeSpecName: "kube-api-access-h5kfk") pod "13b836f2-52ec-4c9c-92ae-f03e0317baeb" (UID: "13b836f2-52ec-4c9c-92ae-f03e0317baeb"). InnerVolumeSpecName "kube-api-access-h5kfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:35.771070 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.771058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b836f2-52ec-4c9c-92ae-f03e0317baeb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13b836f2-52ec-4c9c-92ae-f03e0317baeb" (UID: "13b836f2-52ec-4c9c-92ae-f03e0317baeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:35.869869 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.869787    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kfk\" (UniqueName: \"kubernetes.io/projected/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-api-access-h5kfk\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:17:35.869869 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.869815    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:17:35.869869 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.869825    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13b836f2-52ec-4c9c-92ae-f03e0317baeb-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:17:35.869869 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.869834    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13b836f2-52ec-4c9c-92ae-f03e0317baeb-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:17:35.869869 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:35.869843    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13b836f2-52ec-4c9c-92ae-f03e0317baeb-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:17:36.555539 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.555494    2562 generic.go:356] "Generic (PLEG): container finished" podID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerID="361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24" exitCode=0
Jul 09 18:17:36.555835 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.555553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerDied","Data":"361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24"}
Jul 09 18:17:36.555835 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.555592    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"
Jul 09 18:17:36.555835 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.555609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7" event={"ID":"13b836f2-52ec-4c9c-92ae-f03e0317baeb","Type":"ContainerDied","Data":"c2be5c010888bd553f7e1e3ccdeda40d46cd00619dc7f821388dfb5588c0b266"}
Jul 09 18:17:36.555835 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.555640    2562 scope.go:117] "RemoveContainer" containerID="361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24"
Jul 09 18:17:36.564223 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.564202    2562 scope.go:117] "RemoveContainer" containerID="759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7"
Jul 09 18:17:36.571759 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.571739    2562 scope.go:117] "RemoveContainer" containerID="a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb"
Jul 09 18:17:36.578754 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.578733    2562 scope.go:117] "RemoveContainer" containerID="25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4"
Jul 09 18:17:36.584545 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.584519    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"]
Jul 09 18:17:36.585948 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.585934    2562 scope.go:117] "RemoveContainer" containerID="361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24"
Jul 09 18:17:36.586282 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:17:36.586258    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24\": container with ID starting with 361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24 not found: ID does not exist" containerID="361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24"
Jul 09 18:17:36.586370 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.586287    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24"} err="failed to get container status \"361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24\": rpc error: code = NotFound desc = could not find container \"361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24\": container with ID starting with 361ac4aa49bdc684fd84105d5e6c42f09c2cc59e2855e7d964bf0ad2c0c2db24 not found: ID does not exist"
Jul 09 18:17:36.586370 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.586305    2562 scope.go:117] "RemoveContainer" containerID="759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7"
Jul 09 18:17:36.586540 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:17:36.586524    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7\": container with ID starting with 759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7 not found: ID does not exist" containerID="759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7"
Jul 09 18:17:36.586578 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.586544    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7"} err="failed to get container status \"759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7\": rpc error: code = NotFound desc = could not find container \"759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7\": container with ID starting with 759b8bb9d8c4b959fad52e3bdfa868f698fd0d148d2497d348f7f939b95829e7 not found: ID does not exist"
Jul 09 18:17:36.586578 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.586556    2562 scope.go:117] "RemoveContainer" containerID="a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb"
Jul 09 18:17:36.586772 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:17:36.586752    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb\": container with ID starting with a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb not found: ID does not exist" containerID="a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb"
Jul 09 18:17:36.586845 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.586781    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb"} err="failed to get container status \"a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb\": rpc error: code = NotFound desc = could not find container \"a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb\": container with ID starting with a1bfa9dc7a649dcbb8c38d019c0fde7e62b9a6d19ade2e80a511bae092985aeb not found: ID does not exist"
Jul 09 18:17:36.586845 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.586803    2562 scope.go:117] "RemoveContainer" containerID="25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4"
Jul 09 18:17:36.587024 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:17:36.587009    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4\": container with ID starting with 25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4 not found: ID does not exist" containerID="25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4"
Jul 09 18:17:36.587060 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.587028    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4"} err="failed to get container status \"25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4\": rpc error: code = NotFound desc = could not find container \"25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4\": container with ID starting with 25930b09452e594c973aed9bb716a095032487e9d30242aa05af885caeacb1c4 not found: ID does not exist"
Jul 09 18:17:36.593377 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:36.593358    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-569c9bcb84-g5ss7"]
Jul 09 18:17:37.542861 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:37.542836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:17:37.543419 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:37.543392    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:17:37.543634 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:37.543610    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:38.563523 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:38.563487    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" path="/var/lib/kubelet/pods/13b836f2-52ec-4c9c-92ae-f03e0317baeb/volumes"
Jul 09 18:17:47.543337 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:47.543298    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:17:47.543714 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:47.543608    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:17:57.543806 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:57.543762    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:17:57.544346 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:17:57.544324    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:07.544005 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:07.543955    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:18:07.544524 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:07.544495    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:17.544235 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:17.544178    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:18:17.544648 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:17.544618    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:27.543541 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:27.543443    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:18:27.543963 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:27.543934    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:34.506090 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:34.506055    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:18:34.509277 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:34.509257    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:18:37.543615 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:37.543571    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:18:37.544064 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:37.544041    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:18:47.544734 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:47.544703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:18:47.547110 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:18:47.545106    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:19:00.777411 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:00.777381    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-rcq7l_02a7ed3d-63d6-465f-8165-fe682bedbedd/kserve-container/0.log"
Jul 09 18:19:01.027331 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.027301    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"]
Jul 09 18:19:01.027633 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.027548    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" containerID="cri-o://c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c" gracePeriod=30
Jul 09 18:19:01.027633 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.027578    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" containerID="cri-o://22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3" gracePeriod=30
Jul 09 18:19:01.027633 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.027594    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" containerID="cri-o://dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444" gracePeriod=30
Jul 09 18:19:01.105513 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105486    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"]
Jul 09 18:19:01.105782 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105769    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent"
Jul 09 18:19:01.105824 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105784    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent"
Jul 09 18:19:01.105824 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="storage-initializer"
Jul 09 18:19:01.105824 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105801    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="storage-initializer"
Jul 09 18:19:01.105824 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105822    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container"
Jul 09 18:19:01.105940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105828    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container"
Jul 09 18:19:01.105940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105833    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy"
Jul 09 18:19:01.105940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105838    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy"
Jul 09 18:19:01.105940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105894    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="agent"
Jul 09 18:19:01.105940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105902    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kube-rbac-proxy"
Jul 09 18:19:01.105940 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.105908    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b836f2-52ec-4c9c-92ae-f03e0317baeb" containerName="kserve-container"
Jul 09 18:19:01.108877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.108863    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.113106 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.113070    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 09 18:19:01.113223 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.113118    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 09 18:19:01.122517 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.122492    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"]
Jul 09 18:19:01.122742 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.122721    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kserve-container" containerID="cri-o://67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c" gracePeriod=30
Jul 09 18:19:01.122813 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.122735    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kube-rbac-proxy" containerID="cri-o://92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8" gracePeriod=30
Jul 09 18:19:01.123905 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.123886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"]
Jul 09 18:19:01.142379 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.142353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:01.283765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.283689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.283765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.283723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.283765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.283739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfc22\" (UniqueName: \"kubernetes.io/projected/88f79c8d-148a-436c-be08-569d9cc1538b-kube-api-access-jfc22\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.283944 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.283892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88f79c8d-148a-436c-be08-569d9cc1538b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.283944 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.283935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.289531 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.289495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:01.365273 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.365251    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:19:01.384712 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.384681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.384828 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.384770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.384828 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.384791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.384828 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.384806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfc22\" (UniqueName: \"kubernetes.io/projected/88f79c8d-148a-436c-be08-569d9cc1538b-kube-api-access-jfc22\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.384997 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.384872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88f79c8d-148a-436c-be08-569d9cc1538b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.385311 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:01.385140    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 09 18:19:01.385311 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:01.385212    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls podName:88f79c8d-148a-436c-be08-569d9cc1538b nodeName:}" failed. No retries permitted until 2026-07-09 18:19:01.885191497 +0000 UTC m=+927.855919385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" (UID: "88f79c8d-148a-436c-be08-569d9cc1538b") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 09 18:19:01.385962 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.385939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88f79c8d-148a-436c-be08-569d9cc1538b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.386245 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.386226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.386300 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.386251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.393994 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.393969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfc22\" (UniqueName: \"kubernetes.io/projected/88f79c8d-148a-436c-be08-569d9cc1538b-kube-api-access-jfc22\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.485788 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.485755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-rbac-proxy-sar-config\") pod \"02a7ed3d-63d6-465f-8165-fe682bedbedd\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") "
Jul 09 18:19:01.485948 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.485802    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls\") pod \"02a7ed3d-63d6-465f-8165-fe682bedbedd\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") "
Jul 09 18:19:01.485948 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.485831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5mkx\" (UniqueName: \"kubernetes.io/projected/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-api-access-j5mkx\") pod \"02a7ed3d-63d6-465f-8165-fe682bedbedd\" (UID: \"02a7ed3d-63d6-465f-8165-fe682bedbedd\") "
Jul 09 18:19:01.486181 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.486157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02a7ed3d-63d6-465f-8165-fe682bedbedd" (UID: "02a7ed3d-63d6-465f-8165-fe682bedbedd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:01.488004 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.487977    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02a7ed3d-63d6-465f-8165-fe682bedbedd" (UID: "02a7ed3d-63d6-465f-8165-fe682bedbedd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:01.488004 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.487993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-api-access-j5mkx" (OuterVolumeSpecName: "kube-api-access-j5mkx") pod "02a7ed3d-63d6-465f-8165-fe682bedbedd" (UID: "02a7ed3d-63d6-465f-8165-fe682bedbedd"). InnerVolumeSpecName "kube-api-access-j5mkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:01.584990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.584900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:01.586982 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.586965    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:01.587047 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.586984    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a7ed3d-63d6-465f-8165-fe682bedbedd-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:01.587047 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.586995    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5mkx\" (UniqueName: \"kubernetes.io/projected/02a7ed3d-63d6-465f-8165-fe682bedbedd-kube-api-access-j5mkx\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:01.866477 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866394    2562 generic.go:356] "Generic (PLEG): container finished" podID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerID="92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8" exitCode=2
Jul 09 18:19:01.866477 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866418    2562 generic.go:356] "Generic (PLEG): container finished" podID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerID="67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c" exitCode=2
Jul 09 18:19:01.866477 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866441    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"
Jul 09 18:19:01.866993 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" event={"ID":"02a7ed3d-63d6-465f-8165-fe682bedbedd","Type":"ContainerDied","Data":"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"}
Jul 09 18:19:01.866993 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" event={"ID":"02a7ed3d-63d6-465f-8165-fe682bedbedd","Type":"ContainerDied","Data":"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"}
Jul 09 18:19:01.866993 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l" event={"ID":"02a7ed3d-63d6-465f-8165-fe682bedbedd","Type":"ContainerDied","Data":"9c9d11598dfe9f5a2d25e79779b1341eac44f8ae5d2a9d94c089e96d7ef07804"}
Jul 09 18:19:01.866993 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.866572    2562 scope.go:117] "RemoveContainer" containerID="92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"
Jul 09 18:19:01.869412 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.869388    2562 generic.go:356] "Generic (PLEG): container finished" podID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerID="dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444" exitCode=2
Jul 09 18:19:01.869511 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.869423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerDied","Data":"dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444"}
Jul 09 18:19:01.875575 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.875560    2562 scope.go:117] "RemoveContainer" containerID="67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"
Jul 09 18:19:01.882645 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.882630    2562 scope.go:117] "RemoveContainer" containerID="92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"
Jul 09 18:19:01.882893 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:01.882876    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8\": container with ID starting with 92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8 not found: ID does not exist" containerID="92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"
Jul 09 18:19:01.882950 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.882900    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"} err="failed to get container status \"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8\": rpc error: code = NotFound desc = could not find container \"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8\": container with ID starting with 92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8 not found: ID does not exist"
Jul 09 18:19:01.882950 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.882915    2562 scope.go:117] "RemoveContainer" containerID="67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"
Jul 09 18:19:01.883179 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:01.883164    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c\": container with ID starting with 67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c not found: ID does not exist" containerID="67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"
Jul 09 18:19:01.883222 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.883183    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"} err="failed to get container status \"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c\": rpc error: code = NotFound desc = could not find container \"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c\": container with ID starting with 67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c not found: ID does not exist"
Jul 09 18:19:01.883222 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.883195    2562 scope.go:117] "RemoveContainer" containerID="92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"
Jul 09 18:19:01.883451 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.883430    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8"} err="failed to get container status \"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8\": rpc error: code = NotFound desc = could not find container \"92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8\": container with ID starting with 92c71cbc5419045ce242ce989872bcb72c61954159ff0d240000b390342364d8 not found: ID does not exist"
Jul 09 18:19:01.883509 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.883453    2562 scope.go:117] "RemoveContainer" containerID="67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"
Jul 09 18:19:01.883687 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.883669    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c"} err="failed to get container status \"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c\": rpc error: code = NotFound desc = could not find container \"67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c\": container with ID starting with 67ccceaec6bab0196c714d886155eb29894a4b76f7b6fdbdaed11ecb9de06f7c not found: ID does not exist"
Jul 09 18:19:01.888862 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.888843    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"]
Jul 09 18:19:01.890511 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.890462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:01.891952 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.891928    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-rcq7l"]
Jul 09 18:19:01.893074 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:01.893055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:02.017996 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:02.017962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:19:02.151150 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:02.151126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"]
Jul 09 18:19:02.153399 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:19:02.153370    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f79c8d_148a_436c_be08_569d9cc1538b.slice/crio-c760e29c929790a26452c4ada7a234dbb3bf0fc39c9cbdbe1f1616826c1c4b29 WatchSource:0}: Error finding container c760e29c929790a26452c4ada7a234dbb3bf0fc39c9cbdbe1f1616826c1c4b29: Status 404 returned error can't find the container with id c760e29c929790a26452c4ada7a234dbb3bf0fc39c9cbdbe1f1616826c1c4b29
Jul 09 18:19:02.538873 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:02.538760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 09 18:19:02.562731 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:02.562697    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" path="/var/lib/kubelet/pods/02a7ed3d-63d6-465f-8165-fe682bedbedd/volumes"
Jul 09 18:19:02.875489 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:02.875391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerStarted","Data":"52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154"}
Jul 09 18:19:02.875489 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:02.875457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerStarted","Data":"c760e29c929790a26452c4ada7a234dbb3bf0fc39c9cbdbe1f1616826c1c4b29"}
Jul 09 18:19:05.889474 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:05.889446    2562 generic.go:356] "Generic (PLEG): container finished" podID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerID="c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c" exitCode=0
Jul 09 18:19:05.889873 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:05.889519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerDied","Data":"c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c"}
Jul 09 18:19:07.538865 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:07.538816    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 09 18:19:07.544194 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:07.544167    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:19:07.544499 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:07.544472    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:12.539541 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:12.539501    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 09 18:19:12.539915 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:12.539615    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:19:17.538707 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:17.538662    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 09 18:19:17.543277 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:17.543247    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:19:17.543680 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:17.543652    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:22.538765 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:22.538717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 09 18:19:27.538997 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:27.538950    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 09 18:19:27.543327 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:27.543296    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 09 18:19:27.543452 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:27.543415    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:19:27.543635 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:27.543604    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:19:27.543729 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:27.543716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:19:29.197764 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:29.197736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:19:31.175636 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.175614    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:19:31.197916 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.197892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-rbac-proxy-sar-config\") pod \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") "
Jul 09 18:19:31.198081 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.198031    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-cabundle-cert\") pod \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") "
Jul 09 18:19:31.198405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.198131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-proxy-tls\") pod \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") "
Jul 09 18:19:31.198405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.198168    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnff2\" (UniqueName: \"kubernetes.io/projected/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-api-access-wnff2\") pod \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") "
Jul 09 18:19:31.198405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.198199    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kserve-provision-location\") pod \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\" (UID: \"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c\") "
Jul 09 18:19:31.199011 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.198294    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" (UID: "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:31.205795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.199075    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:31.205795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.199295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" (UID: "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:19:31.205795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.199321    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" (UID: "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:19:31.205795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.201512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" (UID: "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:19:31.207648 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.207621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-api-access-wnff2" (OuterVolumeSpecName: "kube-api-access-wnff2") pod "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" (UID: "c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c"). InnerVolumeSpecName "kube-api-access-wnff2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:19:31.300421 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.300354    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:31.300421 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.300379    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:31.300421 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.300389    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnff2\" (UniqueName: \"kubernetes.io/projected/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kube-api-access-wnff2\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:31.300421 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.300398    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:19:31.985601 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.985565    2562 generic.go:356] "Generic (PLEG): container finished" podID="88f79c8d-148a-436c-be08-569d9cc1538b" containerID="52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154" exitCode=0
Jul 09 18:19:31.985781 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.985644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerDied","Data":"52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154"}
Jul 09 18:19:31.987183 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.987163    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:19:31.988240 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.988208    2562 generic.go:356] "Generic (PLEG): container finished" podID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerID="22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3" exitCode=0
Jul 09 18:19:31.988331 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.988262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerDied","Data":"22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3"}
Jul 09 18:19:31.988331 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.988315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk" event={"ID":"c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c","Type":"ContainerDied","Data":"30c408ca5d217612cee3c86d93f51ce4ba8a698c9d45378960e54e33e8540cfc"}
Jul 09 18:19:31.988401 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.988334    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"
Jul 09 18:19:31.988401 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.988346    2562 scope.go:117] "RemoveContainer" containerID="22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3"
Jul 09 18:19:31.997185 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:31.997169    2562 scope.go:117] "RemoveContainer" containerID="dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444"
Jul 09 18:19:32.003866 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.003848    2562 scope.go:117] "RemoveContainer" containerID="c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c"
Jul 09 18:19:32.011157 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.011135    2562 scope.go:117] "RemoveContainer" containerID="4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc"
Jul 09 18:19:32.018301 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.018279    2562 scope.go:117] "RemoveContainer" containerID="22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3"
Jul 09 18:19:32.018574 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:32.018553    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3\": container with ID starting with 22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3 not found: ID does not exist" containerID="22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3"
Jul 09 18:19:32.018637 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.018585    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3"} err="failed to get container status \"22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3\": rpc error: code = NotFound desc = could not find container \"22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3\": container with ID starting with 22011d53cad068c257d501536c0f2b3bc65fe26991ab62d8f2b69b87348a33b3 not found: ID does not exist"
Jul 09 18:19:32.018637 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.018607    2562 scope.go:117] "RemoveContainer" containerID="dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444"
Jul 09 18:19:32.018864 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:32.018843    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444\": container with ID starting with dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444 not found: ID does not exist" containerID="dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444"
Jul 09 18:19:32.018951 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.018868    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444"} err="failed to get container status \"dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444\": rpc error: code = NotFound desc = could not find container \"dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444\": container with ID starting with dcbe67365e19a66bf9bad821d0feb17c44886c5ec7d9b85d225c11464e109444 not found: ID does not exist"
Jul 09 18:19:32.018951 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.018883    2562 scope.go:117] "RemoveContainer" containerID="c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c"
Jul 09 18:19:32.019159 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:32.019143    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c\": container with ID starting with c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c not found: ID does not exist" containerID="c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c"
Jul 09 18:19:32.019242 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.019167    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c"} err="failed to get container status \"c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c\": rpc error: code = NotFound desc = could not find container \"c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c\": container with ID starting with c6a3b97d7e2024167553c166656344e60d653a1ce13277d908dce21219dd5e2c not found: ID does not exist"
Jul 09 18:19:32.019242 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.019179    2562 scope.go:117] "RemoveContainer" containerID="4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc"
Jul 09 18:19:32.019833 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.019813    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"]
Jul 09 18:19:32.019898 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:19:32.019813    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc\": container with ID starting with 4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc not found: ID does not exist" containerID="4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc"
Jul 09 18:19:32.019898 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.019862    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc"} err="failed to get container status \"4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc\": rpc error: code = NotFound desc = could not find container \"4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc\": container with ID starting with 4cfad2394589b25b24a3a09529814f15ca84e688858211a7cf00ecfbbb1eeecc not found: ID does not exist"
Jul 09 18:19:32.029146 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.029125    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57ff67ccd6-d7gbk"]
Jul 09 18:19:32.567026 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:19:32.566454    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" path="/var/lib/kubelet/pods/c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c/volumes"
Jul 09 18:20:02.164441 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:20:02.164382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:20:02.313362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:20:02.313312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:21:55.569667 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:55.569633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerStarted","Data":"57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99"}
Jul 09 18:21:55.570077 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:55.569686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerStarted","Data":"9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded"}
Jul 09 18:21:55.570077 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:55.569830    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:21:55.602289 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:55.602246    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" podStartSLOduration=2.524500116 podStartE2EDuration="2m54.602232531s" totalImagesPullingTime="2m22.768647112s" totalInitContainerRuntime="29.309085303s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:55.600968495 +0000 UTC m=+1101.571696395" watchObservedRunningTime="2026-07-09 18:21:55.602232531 +0000 UTC m=+1101.572960430"
Jul 09 18:21:56.573308 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:56.573266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:21:56.574482 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:56.574453    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 18:21:57.577793 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:21:57.577748    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 18:22:02.582321 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:02.582287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:22:02.582906 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:02.582872    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 09 18:22:12.584252 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:12.584217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:22:21.251763 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:21.251727    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"]
Jul 09 18:22:21.252234 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:21.251992    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container" containerID="cri-o://9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded" gracePeriod=30
Jul 09 18:22:21.252234 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:21.252087    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kube-rbac-proxy" containerID="cri-o://57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99" gracePeriod=30
Jul 09 18:22:21.316508 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:21.316479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:21.667534 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:21.667501    2562 generic.go:356] "Generic (PLEG): container finished" podID="88f79c8d-148a-436c-be08-569d9cc1538b" containerID="57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99" exitCode=2
Jul 09 18:22:21.667691 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:21.667576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerDied","Data":"57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99"}
Jul 09 18:22:22.233895 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.233822    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:22:22.338505 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.338468    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfc22\" (UniqueName: \"kubernetes.io/projected/88f79c8d-148a-436c-be08-569d9cc1538b-kube-api-access-jfc22\") pod \"88f79c8d-148a-436c-be08-569d9cc1538b\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") "
Jul 09 18:22:22.338990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.338509    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls\") pod \"88f79c8d-148a-436c-be08-569d9cc1538b\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") "
Jul 09 18:22:22.338990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.338574    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-kube-rbac-proxy-sar-config\") pod \"88f79c8d-148a-436c-be08-569d9cc1538b\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") "
Jul 09 18:22:22.338990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.338673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88f79c8d-148a-436c-be08-569d9cc1538b-kserve-provision-location\") pod \"88f79c8d-148a-436c-be08-569d9cc1538b\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") "
Jul 09 18:22:22.338990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.338707    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-cabundle-cert\") pod \"88f79c8d-148a-436c-be08-569d9cc1538b\" (UID: \"88f79c8d-148a-436c-be08-569d9cc1538b\") "
Jul 09 18:22:22.338990 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.338946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88f79c8d-148a-436c-be08-569d9cc1538b" (UID: "88f79c8d-148a-436c-be08-569d9cc1538b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:22:22.339445 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.339123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88f79c8d-148a-436c-be08-569d9cc1538b" (UID: "88f79c8d-148a-436c-be08-569d9cc1538b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:22:22.340779 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.340755    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88f79c8d-148a-436c-be08-569d9cc1538b" (UID: "88f79c8d-148a-436c-be08-569d9cc1538b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:22:22.341076 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.341056    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f79c8d-148a-436c-be08-569d9cc1538b-kube-api-access-jfc22" (OuterVolumeSpecName: "kube-api-access-jfc22") pod "88f79c8d-148a-436c-be08-569d9cc1538b" (UID: "88f79c8d-148a-436c-be08-569d9cc1538b"). InnerVolumeSpecName "kube-api-access-jfc22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:22:22.344435 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.344414    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f79c8d-148a-436c-be08-569d9cc1538b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88f79c8d-148a-436c-be08-569d9cc1538b" (UID: "88f79c8d-148a-436c-be08-569d9cc1538b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:22:22.440240 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.440204    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88f79c8d-148a-436c-be08-569d9cc1538b-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:22:22.440240 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.440234    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:22:22.440240 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.440247    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfc22\" (UniqueName: \"kubernetes.io/projected/88f79c8d-148a-436c-be08-569d9cc1538b-kube-api-access-jfc22\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:22:22.440463 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.440257    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88f79c8d-148a-436c-be08-569d9cc1538b-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:22:22.440463 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.440268    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88f79c8d-148a-436c-be08-569d9cc1538b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:22:22.673040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.672998    2562 generic.go:356] "Generic (PLEG): container finished" podID="88f79c8d-148a-436c-be08-569d9cc1538b" containerID="9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded" exitCode=0
Jul 09 18:22:22.673221 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.673080    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"
Jul 09 18:22:22.673221 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.673112    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerDied","Data":"9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded"}
Jul 09 18:22:22.673221 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.673187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s" event={"ID":"88f79c8d-148a-436c-be08-569d9cc1538b","Type":"ContainerDied","Data":"c760e29c929790a26452c4ada7a234dbb3bf0fc39c9cbdbe1f1616826c1c4b29"}
Jul 09 18:22:22.673322 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.673225    2562 scope.go:117] "RemoveContainer" containerID="57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99"
Jul 09 18:22:22.681248 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.681228    2562 scope.go:117] "RemoveContainer" containerID="9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded"
Jul 09 18:22:22.688169 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.688125    2562 scope.go:117] "RemoveContainer" containerID="52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154"
Jul 09 18:22:22.693435 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.693413    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"]
Jul 09 18:22:22.695655 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.695628    2562 scope.go:117] "RemoveContainer" containerID="57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99"
Jul 09 18:22:22.695951 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:22:22.695929    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99\": container with ID starting with 57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99 not found: ID does not exist" containerID="57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99"
Jul 09 18:22:22.696015 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.695958    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99"} err="failed to get container status \"57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99\": rpc error: code = NotFound desc = could not find container \"57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99\": container with ID starting with 57d12627ea68b54d341d6acb6cdec312820b83dab4b921a263358a65b3f0da99 not found: ID does not exist"
Jul 09 18:22:22.696015 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.695978    2562 scope.go:117] "RemoveContainer" containerID="9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded"
Jul 09 18:22:22.696389 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:22:22.696367    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded\": container with ID starting with 9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded not found: ID does not exist" containerID="9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded"
Jul 09 18:22:22.696480 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.696394    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded"} err="failed to get container status \"9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded\": rpc error: code = NotFound desc = could not find container \"9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded\": container with ID starting with 9ee0663a53082dfcd4c5c9d35ba989999e63d37983c741c81850770615e49ded not found: ID does not exist"
Jul 09 18:22:22.696480 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.696415    2562 scope.go:117] "RemoveContainer" containerID="52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154"
Jul 09 18:22:22.696807 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:22:22.696784    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154\": container with ID starting with 52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154 not found: ID does not exist" containerID="52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154"
Jul 09 18:22:22.696913 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.696813    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154"} err="failed to get container status \"52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154\": rpc error: code = NotFound desc = could not find container \"52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154\": container with ID starting with 52ed8f7c6d69b200eb81d0210d6425b5b83fefef053b0b35278f734767369154 not found: ID does not exist"
Jul 09 18:22:22.697893 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:22.697876    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-h4w6s"]
Jul 09 18:22:24.562958 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:24.562926    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" path="/var/lib/kubelet/pods/88f79c8d-148a-436c-be08-569d9cc1538b/volumes"
Jul 09 18:22:26.601901 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.601866    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:26.681141 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681086    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"]
Jul 09 18:22:26.681361 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681349    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681363    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681373    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681379    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681390    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681397    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681404    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kserve-container"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681409    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kserve-container"
Jul 09 18:22:26.681415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681415    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="storage-initializer"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681421    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="storage-initializer"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681427    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681432    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681440    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="storage-initializer"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681445    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="storage-initializer"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681451    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681456    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681462    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681466    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681504    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681512    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kserve-container"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681519    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681525    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kserve-container"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681531    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f79c8d-148a-436c-be08-569d9cc1538b" containerName="kube-rbac-proxy"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681536    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a7ed3d-63d6-465f-8165-fe682bedbedd" containerName="kserve-container"
Jul 09 18:22:26.681677 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.681540    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c348671f-a7ef-46ad-9b57-5b5ac2dd5a5c" containerName="agent"
Jul 09 18:22:26.724081 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.724039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"]
Jul 09 18:22:26.724275 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.724203    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.728950 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.728918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 09 18:22:26.729136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.728977    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:22:26.729136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.729009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:22:26.730804 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.730479    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 09 18:22:26.730804 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.730494    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:22:26.731620 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.731603    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xqd54\""
Jul 09 18:22:26.731718 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.731630    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:22:26.870476 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.870370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmkk8\" (UniqueName: \"kubernetes.io/projected/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-api-access-mmkk8\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.870476 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.870418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.870698 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.870507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.870758 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.870534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.871039 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.871011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.965057 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.965020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:22:26.972126 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.972075    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.972126 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.972128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.972344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.972158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.972344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.972181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmkk8\" (UniqueName: \"kubernetes.io/projected/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-api-access-mmkk8\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.972344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.972201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.972620 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.972601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.973028 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.973004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.973068 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.973042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.974710 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.974691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:26.983369 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:26.983336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmkk8\" (UniqueName: \"kubernetes.io/projected/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-api-access-mmkk8\") pod \"isvc-autogluon-v2-predictor-67b849d456-6lb66\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:27.034211 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:27.034174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:27.192448 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:27.192268    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"]
Jul 09 18:22:27.195218 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:22:27.195187    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbde0535_cadf_4f1c_9daa_6db0c87e1ec8.slice/crio-65161d2b583e672f97cfd69407a89f1ba113b869b76d36e17559b78602c65ff9 WatchSource:0}: Error finding container 65161d2b583e672f97cfd69407a89f1ba113b869b76d36e17559b78602c65ff9: Status 404 returned error can't find the container with id 65161d2b583e672f97cfd69407a89f1ba113b869b76d36e17559b78602c65ff9
Jul 09 18:22:27.693304 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:27.693270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerStarted","Data":"5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3"}
Jul 09 18:22:27.693674 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:27.693331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerStarted","Data":"65161d2b583e672f97cfd69407a89f1ba113b869b76d36e17559b78602c65ff9"}
Jul 09 18:22:32.713725 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:32.713691    2562 generic.go:356] "Generic (PLEG): container finished" podID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerID="5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3" exitCode=0
Jul 09 18:22:32.713725 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:32.713728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerDied","Data":"5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3"}
Jul 09 18:22:33.721002 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:33.720966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerStarted","Data":"b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965"}
Jul 09 18:22:33.721432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:33.721053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerStarted","Data":"7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c"}
Jul 09 18:22:33.721432 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:33.721294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:33.721543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:33.721454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:33.755147 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:33.755070    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" podStartSLOduration=2.9269458569999998 podStartE2EDuration="7.75505373s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.828107873s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:22:33.752726312 +0000 UTC m=+1139.723454210" watchObservedRunningTime="2026-07-09 18:22:33.75505373 +0000 UTC m=+1139.725781629"
Jul 09 18:22:39.732776 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:39.732744    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:22:59.838268 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:22:59.838231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:23:34.528971 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:23:34.528928    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:23:34.529462 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:23:34.529389    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:24:11.562418 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:11.562387    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:24:16.744834 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:16.744803    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"]
Jul 09 18:24:16.745263 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:16.745051    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kserve-container" containerID="cri-o://7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c" gracePeriod=30
Jul 09 18:24:16.745263 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:16.745112    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kube-rbac-proxy" containerID="cri-o://b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965" gracePeriod=30
Jul 09 18:24:16.768852 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:16.768826    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:17.103735 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.103636    2562 generic.go:356] "Generic (PLEG): container finished" podID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerID="b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965" exitCode=2
Jul 09 18:24:17.103735 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.103716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerDied","Data":"b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965"}
Jul 09 18:24:17.596453 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.596430    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:24:17.681205 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681175    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kserve-provision-location\") pod \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") "
Jul 09 18:24:17.681374 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681233    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-proxy-tls\") pod \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") "
Jul 09 18:24:17.681374 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681270    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-cabundle-cert\") pod \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") "
Jul 09 18:24:17.681374 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-rbac-proxy-sar-config\") pod \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") "
Jul 09 18:24:17.681374 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmkk8\" (UniqueName: \"kubernetes.io/projected/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-api-access-mmkk8\") pod \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\" (UID: \"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8\") "
Jul 09 18:24:17.681690 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" (UID: "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:17.681766 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.681666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" (UID: "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:24:17.683551 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.683525    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" (UID: "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:17.683664 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.683564    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-api-access-mmkk8" (OuterVolumeSpecName: "kube-api-access-mmkk8") pod "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" (UID: "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8"). InnerVolumeSpecName "kube-api-access-mmkk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:17.686146 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.686119    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" (UID: "fbde0535-cadf-4f1c-9daa-6db0c87e1ec8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:17.782500 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.782424    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:24:17.782500 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.782459    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:24:17.782500 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.782471    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:24:17.782500 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.782479    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:24:17.782500 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:17.782488    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmkk8\" (UniqueName: \"kubernetes.io/projected/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8-kube-api-access-mmkk8\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:24:18.109122 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.109017    2562 generic.go:356] "Generic (PLEG): container finished" podID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerID="7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c" exitCode=0
Jul 09 18:24:18.109122 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.109070    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerDied","Data":"7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c"}
Jul 09 18:24:18.109295 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.109124    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"
Jul 09 18:24:18.109295 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.109194    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66" event={"ID":"fbde0535-cadf-4f1c-9daa-6db0c87e1ec8","Type":"ContainerDied","Data":"65161d2b583e672f97cfd69407a89f1ba113b869b76d36e17559b78602c65ff9"}
Jul 09 18:24:18.109295 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.109236    2562 scope.go:117] "RemoveContainer" containerID="b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965"
Jul 09 18:24:18.118282 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.118265    2562 scope.go:117] "RemoveContainer" containerID="7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c"
Jul 09 18:24:18.125116 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.125081    2562 scope.go:117] "RemoveContainer" containerID="5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3"
Jul 09 18:24:18.131943 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.131923    2562 scope.go:117] "RemoveContainer" containerID="b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965"
Jul 09 18:24:18.132295 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:24:18.132274    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965\": container with ID starting with b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965 not found: ID does not exist" containerID="b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965"
Jul 09 18:24:18.132362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.132304    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965"} err="failed to get container status \"b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965\": rpc error: code = NotFound desc = could not find container \"b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965\": container with ID starting with b9eefc9014ccc82f0b824e4d033d58a389974a9364d519918f4cda196db97965 not found: ID does not exist"
Jul 09 18:24:18.132362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.132326    2562 scope.go:117] "RemoveContainer" containerID="7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c"
Jul 09 18:24:18.132596 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:24:18.132577    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c\": container with ID starting with 7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c not found: ID does not exist" containerID="7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c"
Jul 09 18:24:18.132644 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.132601    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c"} err="failed to get container status \"7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c\": rpc error: code = NotFound desc = could not find container \"7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c\": container with ID starting with 7b3b8879a6973f2910899af404701a8b518dd8bbeb2d3c0428a7ef8c44775d0c not found: ID does not exist"
Jul 09 18:24:18.132644 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.132617    2562 scope.go:117] "RemoveContainer" containerID="5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3"
Jul 09 18:24:18.132879 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.132858    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"]
Jul 09 18:24:18.132917 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:24:18.132861    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3\": container with ID starting with 5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3 not found: ID does not exist" containerID="5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3"
Jul 09 18:24:18.132953 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.132909    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3"} err="failed to get container status \"5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3\": rpc error: code = NotFound desc = could not find container \"5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3\": container with ID starting with 5c5621d0d75ab3c62ab67a11464402501b222f22fd8b9a099cde42b22a87a6e3 not found: ID does not exist"
Jul 09 18:24:18.136883 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.136863    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-6lb66"]
Jul 09 18:24:18.562344 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:18.562309    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" path="/var/lib/kubelet/pods/fbde0535-cadf-4f1c-9daa-6db0c87e1ec8/volumes"
Jul 09 18:24:21.887108 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.887060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:21.948111 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948075    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"]
Jul 09 18:24:21.948335 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948322    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="storage-initializer"
Jul 09 18:24:21.948382 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948337    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="storage-initializer"
Jul 09 18:24:21.948382 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948355    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kserve-container"
Jul 09 18:24:21.948382 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948360    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kserve-container"
Jul 09 18:24:21.948382 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948369    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kube-rbac-proxy"
Jul 09 18:24:21.948382 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kube-rbac-proxy"
Jul 09 18:24:21.948524 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948429    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kserve-container"
Jul 09 18:24:21.948524 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.948437    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbde0535-cadf-4f1c-9daa-6db0c87e1ec8" containerName="kube-rbac-proxy"
Jul 09 18:24:21.950724 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.950695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:24:21.952963 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.952947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:21.955824 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.955807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:24:21.955824 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.955812    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:24:21.955965 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.955812    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 09 18:24:21.957314 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.957284    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xqd54\""
Jul 09 18:24:21.957314 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.957284    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:24:21.957471 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.957287    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 09 18:24:21.957471 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.957317    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:24:21.962189 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:21.962171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"]
Jul 09 18:24:22.111992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.111946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nmv\" (UniqueName: \"kubernetes.io/projected/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-api-access-75nmv\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.111992 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.111997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.112261 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.112025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.112261 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.112052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.112261 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.112090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213150 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213150 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nmv\" (UniqueName: \"kubernetes.io/projected/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-api-access-75nmv\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213150 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213398 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213398 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213590 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213809 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.213931 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.213854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.215827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.215802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.224321 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.224300    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nmv\" (UniqueName: \"kubernetes.io/projected/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-api-access-75nmv\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.264030 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.263995    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:22.393043 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:22.393017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"]
Jul 09 18:24:22.395396 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:24:22.395369    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf60e1d_8be0_4c96_a7d6_2f359ac7f995.slice/crio-085734d72207d052f8b4cf57aba54aa41294d2887f0320e40fddb47ea7e5730e WatchSource:0}: Error finding container 085734d72207d052f8b4cf57aba54aa41294d2887f0320e40fddb47ea7e5730e: Status 404 returned error can't find the container with id 085734d72207d052f8b4cf57aba54aa41294d2887f0320e40fddb47ea7e5730e
Jul 09 18:24:23.129135 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:23.129086    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerStarted","Data":"cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d"}
Jul 09 18:24:23.129537 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:23.129190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerStarted","Data":"085734d72207d052f8b4cf57aba54aa41294d2887f0320e40fddb47ea7e5730e"}
Jul 09 18:24:53.237164 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:53.237131    2562 generic.go:356] "Generic (PLEG): container finished" podID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerID="cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d" exitCode=0
Jul 09 18:24:53.237578 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:53.237212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerDied","Data":"cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d"}
Jul 09 18:24:54.243676 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:54.243645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerStarted","Data":"c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6"}
Jul 09 18:24:54.244186 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:54.243718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerStarted","Data":"d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57"}
Jul 09 18:24:54.244186 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:54.243952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:54.244186 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:54.243996    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:24:54.270175 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:24:54.270121    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" podStartSLOduration=3.321581261 podStartE2EDuration="33.270086971s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.94850571s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:24:54.267695996 +0000 UTC m=+1280.238423896" watchObservedRunningTime="2026-07-09 18:24:54.270086971 +0000 UTC m=+1280.240814872"
Jul 09 18:25:00.254140 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:25:00.254110    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:26:32.563388 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:32.563358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:26:42.172637 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.172607    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"]
Jul 09 18:26:42.173117 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.172825    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kserve-container" containerID="cri-o://d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57" gracePeriod=30
Jul 09 18:26:42.173117 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.172872    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kube-rbac-proxy" containerID="cri-o://c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6" gracePeriod=30
Jul 09 18:26:42.261373 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.261336    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:42.558662 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.558557    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 09 18:26:42.638513 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.638478    2562 generic.go:356] "Generic (PLEG): container finished" podID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerID="c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6" exitCode=2
Jul 09 18:26:42.638675 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.638551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerDied","Data":"c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6"}
Jul 09 18:26:42.916952 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.916932    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:26:42.939026 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939006    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-rbac-proxy-sar-config\") pod \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") "
Jul 09 18:26:42.939137 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939040    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-cabundle-cert\") pod \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") "
Jul 09 18:26:42.939137 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939073    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-proxy-tls\") pod \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") "
Jul 09 18:26:42.939244 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75nmv\" (UniqueName: \"kubernetes.io/projected/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-api-access-75nmv\") pod \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") "
Jul 09 18:26:42.939244 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939172    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kserve-provision-location\") pod \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\" (UID: \"daf60e1d-8be0-4c96-a7d6-2f359ac7f995\") "
Jul 09 18:26:42.939438 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daf60e1d-8be0-4c96-a7d6-2f359ac7f995" (UID: "daf60e1d-8be0-4c96-a7d6-2f359ac7f995"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:42.939522 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.939456    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "daf60e1d-8be0-4c96-a7d6-2f359ac7f995" (UID: "daf60e1d-8be0-4c96-a7d6-2f359ac7f995"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:42.941147 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.941119    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daf60e1d-8be0-4c96-a7d6-2f359ac7f995" (UID: "daf60e1d-8be0-4c96-a7d6-2f359ac7f995"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:42.941268 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.941179    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-api-access-75nmv" (OuterVolumeSpecName: "kube-api-access-75nmv") pod "daf60e1d-8be0-4c96-a7d6-2f359ac7f995" (UID: "daf60e1d-8be0-4c96-a7d6-2f359ac7f995"). InnerVolumeSpecName "kube-api-access-75nmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:42.943819 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:42.943792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daf60e1d-8be0-4c96-a7d6-2f359ac7f995" (UID: "daf60e1d-8be0-4c96-a7d6-2f359ac7f995"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:26:43.040470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.040448    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.040470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.040469    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.040595 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.040479    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.040595 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.040486    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75nmv\" (UniqueName: \"kubernetes.io/projected/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kube-api-access-75nmv\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.040595 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.040495    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf60e1d-8be0-4c96-a7d6-2f359ac7f995-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:26:43.643561 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.643521    2562 generic.go:356] "Generic (PLEG): container finished" podID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerID="d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57" exitCode=0
Jul 09 18:26:43.643981 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.643572    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"
Jul 09 18:26:43.643981 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.643569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerDied","Data":"d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57"}
Jul 09 18:26:43.643981 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.643627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd" event={"ID":"daf60e1d-8be0-4c96-a7d6-2f359ac7f995","Type":"ContainerDied","Data":"085734d72207d052f8b4cf57aba54aa41294d2887f0320e40fddb47ea7e5730e"}
Jul 09 18:26:43.643981 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.643655    2562 scope.go:117] "RemoveContainer" containerID="c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6"
Jul 09 18:26:43.652048 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.652034    2562 scope.go:117] "RemoveContainer" containerID="d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57"
Jul 09 18:26:43.659126 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.659111    2562 scope.go:117] "RemoveContainer" containerID="cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d"
Jul 09 18:26:43.666069 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666055    2562 scope.go:117] "RemoveContainer" containerID="c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6"
Jul 09 18:26:43.666346 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:26:43.666321    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6\": container with ID starting with c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6 not found: ID does not exist" containerID="c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6"
Jul 09 18:26:43.666403 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666346    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6"} err="failed to get container status \"c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6\": rpc error: code = NotFound desc = could not find container \"c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6\": container with ID starting with c3bbf58c0cc00c04fed990b3580a0d137d7b023da1cdc4859e8f3ac54beeb8e6 not found: ID does not exist"
Jul 09 18:26:43.666403 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666363    2562 scope.go:117] "RemoveContainer" containerID="d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57"
Jul 09 18:26:43.666586 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:26:43.666563    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57\": container with ID starting with d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57 not found: ID does not exist" containerID="d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57"
Jul 09 18:26:43.666630 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666583    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57"} err="failed to get container status \"d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57\": rpc error: code = NotFound desc = could not find container \"d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57\": container with ID starting with d31f7a53d74e111b735124cc13591e1e8e4596552e2ccb0767de7329c2150d57 not found: ID does not exist"
Jul 09 18:26:43.666630 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666595    2562 scope.go:117] "RemoveContainer" containerID="cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d"
Jul 09 18:26:43.666630 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666607    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"]
Jul 09 18:26:43.666855 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:26:43.666839    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d\": container with ID starting with cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d not found: ID does not exist" containerID="cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d"
Jul 09 18:26:43.666891 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.666862    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d"} err="failed to get container status \"cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d\": rpc error: code = NotFound desc = could not find container \"cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d\": container with ID starting with cb9c7356bff49c0e4a2365d96e55c2baec175b3731903f5d20d2fda57d599d7d not found: ID does not exist"
Jul 09 18:26:43.670959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:43.670942    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-4sbgd"]
Jul 09 18:26:44.562534 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:44.562501    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" path="/var/lib/kubelet/pods/daf60e1d-8be0-4c96-a7d6-2f359ac7f995/volumes"
Jul 09 18:26:47.407469 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.407432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:47.537702 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537675    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"]
Jul 09 18:26:47.537917 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537906    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kserve-container"
Jul 09 18:26:47.537959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537918    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kserve-container"
Jul 09 18:26:47.537959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537934    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="storage-initializer"
Jul 09 18:26:47.537959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537940    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="storage-initializer"
Jul 09 18:26:47.537959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537946    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kube-rbac-proxy"
Jul 09 18:26:47.537959 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537952    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kube-rbac-proxy"
Jul 09 18:26:47.538122 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537988    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kserve-container"
Jul 09 18:26:47.538122 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.537996    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf60e1d-8be0-4c96-a7d6-2f359ac7f995" containerName="kube-rbac-proxy"
Jul 09 18:26:47.542718 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.542689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.545755 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.545726    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:26:47.545878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.545734    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 09 18:26:47.545878 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.545823    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 09 18:26:47.545970 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.545898    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:26:47.547196 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.547178    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:26:47.547253 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.547198    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:26:47.547293 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.547256    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xqd54\""
Jul 09 18:26:47.560986 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.560962    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"]
Jul 09 18:26:47.569583 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.569561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7643940-45d1-4d53-8c82-6eea4ea31ce0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.569676 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.569657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.569715 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.569682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.569715 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.569702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.569784 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.569721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qz6b\" (UniqueName: \"kubernetes.io/projected/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-api-access-2qz6b\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.634288 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.634265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:26:47.670057 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.669997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670057 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670057 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670278 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qz6b\" (UniqueName: \"kubernetes.io/projected/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-api-access-2qz6b\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670278 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7643940-45d1-4d53-8c82-6eea4ea31ce0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670493 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670720 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.670876 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.670855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.672605 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.672583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7643940-45d1-4d53-8c82-6eea4ea31ce0-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.680810 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.680782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qz6b\" (UniqueName: \"kubernetes.io/projected/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-api-access-2qz6b\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.852656 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.852624    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:47.986621 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:47.986590    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"]
Jul 09 18:26:47.989856 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:26:47.989830    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7643940_45d1_4d53_8c82_6eea4ea31ce0.slice/crio-24cd8b67e6e23f2af77de096bbde1c27b975ebc77ea19925208bd7f612b25601 WatchSource:0}: Error finding container 24cd8b67e6e23f2af77de096bbde1c27b975ebc77ea19925208bd7f612b25601: Status 404 returned error can't find the container with id 24cd8b67e6e23f2af77de096bbde1c27b975ebc77ea19925208bd7f612b25601
Jul 09 18:26:48.663138 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:48.663081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerStarted","Data":"3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5"}
Jul 09 18:26:48.663517 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:48.663173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerStarted","Data":"24cd8b67e6e23f2af77de096bbde1c27b975ebc77ea19925208bd7f612b25601"}
Jul 09 18:26:52.680511 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:52.680472    2562 generic.go:356] "Generic (PLEG): container finished" podID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerID="3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5" exitCode=0
Jul 09 18:26:52.680892 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:52.680544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerDied","Data":"3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5"}
Jul 09 18:26:53.687270 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:53.687239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerStarted","Data":"9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478"}
Jul 09 18:26:53.687643 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:53.687296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerStarted","Data":"d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1"}
Jul 09 18:26:53.687643 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:53.687477    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:53.709858 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:53.709805    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" podStartSLOduration=2.147406115 podStartE2EDuration="6.709791233s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.562385118s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:53.708966017 +0000 UTC m=+1399.679693927" watchObservedRunningTime="2026-07-09 18:26:53.709791233 +0000 UTC m=+1399.680519131"
Jul 09 18:26:54.691013 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:54.690979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:26:57.326237 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:26:57.326201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:27:00.700963 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:27:00.700936    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:28:27.561856 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:27.561828    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:28:34.553392 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:34.553363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:28:34.553827 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:34.553473    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:28:37.535411 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:37.535377    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"]
Jul 09 18:28:37.535887 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:37.535618    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kserve-container" containerID="cri-o://d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1" gracePeriod=30
Jul 09 18:28:37.535887 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:37.535700    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kube-rbac-proxy" containerID="cri-o://9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478" gracePeriod=30
Jul 09 18:28:37.614172 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:37.614139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:38.064661 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.064628    2562 generic.go:356] "Generic (PLEG): container finished" podID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerID="9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478" exitCode=2
Jul 09 18:28:38.064831 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.064672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerDied","Data":"9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478"}
Jul 09 18:28:38.373482 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.373455    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:28:38.478730 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.478699    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qz6b\" (UniqueName: \"kubernetes.io/projected/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-api-access-2qz6b\") pod \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") "
Jul 09 18:28:38.478877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.478758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kserve-provision-location\") pod \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") "
Jul 09 18:28:38.478877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.478787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7643940-45d1-4d53-8c82-6eea4ea31ce0-proxy-tls\") pod \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") "
Jul 09 18:28:38.478877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.478810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-cabundle-cert\") pod \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") "
Jul 09 18:28:38.478877 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.478834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-rbac-proxy-sar-config\") pod \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\" (UID: \"e7643940-45d1-4d53-8c82-6eea4ea31ce0\") "
Jul 09 18:28:38.479218 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.479191    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7643940-45d1-4d53-8c82-6eea4ea31ce0" (UID: "e7643940-45d1-4d53-8c82-6eea4ea31ce0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:38.479334 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.479240    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7643940-45d1-4d53-8c82-6eea4ea31ce0" (UID: "e7643940-45d1-4d53-8c82-6eea4ea31ce0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:38.480980 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.480954    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7643940-45d1-4d53-8c82-6eea4ea31ce0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7643940-45d1-4d53-8c82-6eea4ea31ce0" (UID: "e7643940-45d1-4d53-8c82-6eea4ea31ce0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:38.481067 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.481032    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-api-access-2qz6b" (OuterVolumeSpecName: "kube-api-access-2qz6b") pod "e7643940-45d1-4d53-8c82-6eea4ea31ce0" (UID: "e7643940-45d1-4d53-8c82-6eea4ea31ce0"). InnerVolumeSpecName "kube-api-access-2qz6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:38.483496 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.483475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7643940-45d1-4d53-8c82-6eea4ea31ce0" (UID: "e7643940-45d1-4d53-8c82-6eea4ea31ce0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:28:38.580174 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.580150    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:28:38.580174 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.580172    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:28:38.580474 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.580184    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qz6b\" (UniqueName: \"kubernetes.io/projected/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kube-api-access-2qz6b\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:28:38.580474 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.580193    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7643940-45d1-4d53-8c82-6eea4ea31ce0-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:28:38.580474 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:38.580202    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7643940-45d1-4d53-8c82-6eea4ea31ce0-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:28:39.070439 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.070407    2562 generic.go:356] "Generic (PLEG): container finished" podID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerID="d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1" exitCode=0
Jul 09 18:28:39.070802 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.070481    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerDied","Data":"d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1"}
Jul 09 18:28:39.070802 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.070528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr" event={"ID":"e7643940-45d1-4d53-8c82-6eea4ea31ce0","Type":"ContainerDied","Data":"24cd8b67e6e23f2af77de096bbde1c27b975ebc77ea19925208bd7f612b25601"}
Jul 09 18:28:39.070802 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.070552    2562 scope.go:117] "RemoveContainer" containerID="9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478"
Jul 09 18:28:39.070802 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.070481    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"
Jul 09 18:28:39.078434 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.078416    2562 scope.go:117] "RemoveContainer" containerID="d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1"
Jul 09 18:28:39.085348 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.085333    2562 scope.go:117] "RemoveContainer" containerID="3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5"
Jul 09 18:28:39.091698 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.091669    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"]
Jul 09 18:28:39.092539 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.092508    2562 scope.go:117] "RemoveContainer" containerID="9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478"
Jul 09 18:28:39.092805 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:28:39.092779    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478\": container with ID starting with 9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478 not found: ID does not exist" containerID="9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478"
Jul 09 18:28:39.092888 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.092817    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478"} err="failed to get container status \"9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478\": rpc error: code = NotFound desc = could not find container \"9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478\": container with ID starting with 9de144e9717529136f623f073cca2286899d544982d1c91fe56be2ff4f54e478 not found: ID does not exist"
Jul 09 18:28:39.092888 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.092842    2562 scope.go:117] "RemoveContainer" containerID="d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1"
Jul 09 18:28:39.093137 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:28:39.093081    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1\": container with ID starting with d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1 not found: ID does not exist" containerID="d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1"
Jul 09 18:28:39.093183 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.093160    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1"} err="failed to get container status \"d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1\": rpc error: code = NotFound desc = could not find container \"d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1\": container with ID starting with d5e588aa5ba949466eb96c684b2e9b07b6747f9c7d95332b28c3439c1d5f4cb1 not found: ID does not exist"
Jul 09 18:28:39.093183 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.093175    2562 scope.go:117] "RemoveContainer" containerID="3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5"
Jul 09 18:28:39.093409 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:28:39.093392    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5\": container with ID starting with 3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5 not found: ID does not exist" containerID="3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5"
Jul 09 18:28:39.093453 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.093414    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5"} err="failed to get container status \"3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5\": rpc error: code = NotFound desc = could not find container \"3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5\": container with ID starting with 3088035dc7a5af52e42ac48cc0adfcec623c98e0fec9bf8a808145479c01b8b5 not found: ID does not exist"
Jul 09 18:28:39.097993 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:39.097974    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cdjkr"]
Jul 09 18:28:40.562493 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:40.562458    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" path="/var/lib/kubelet/pods/e7643940-45d1-4d53-8c82-6eea4ea31ce0/volumes"
Jul 09 18:28:42.660654 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.660616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:42.757968 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.757942    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"]
Jul 09 18:28:42.758259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758187    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="storage-initializer"
Jul 09 18:28:42.758259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758197    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="storage-initializer"
Jul 09 18:28:42.758259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758205    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kserve-container"
Jul 09 18:28:42.758259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758211    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kserve-container"
Jul 09 18:28:42.758259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758225    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kube-rbac-proxy"
Jul 09 18:28:42.758259 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758231    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kube-rbac-proxy"
Jul 09 18:28:42.758457 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758274    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kube-rbac-proxy"
Jul 09 18:28:42.758457 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.758282    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7643940-45d1-4d53-8c82-6eea4ea31ce0" containerName="kserve-container"
Jul 09 18:28:42.762815 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.762795    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.765926 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.765903    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:28:42.766025 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.765906    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 18:28:42.766228 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.766212    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 18:28:42.766313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.766282    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 09 18:28:42.766313 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.766304    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:28:42.767439 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.767425    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xqd54\""
Jul 09 18:28:42.767541 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.767520    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 09 18:28:42.782683 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.782661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"]
Jul 09 18:28:42.810171 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.810145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcfc\" (UniqueName: \"kubernetes.io/projected/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-api-access-2mcfc\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.810287 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.810214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.810287 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.810251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19787062-9e22-44ed-b92f-46e8ddd9410b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.810392 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.810313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19787062-9e22-44ed-b92f-46e8ddd9410b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.810392 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.810346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.833714 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.833691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:28:42.910859 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.910833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19787062-9e22-44ed-b92f-46e8ddd9410b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.910939 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.910870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.910939 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.910893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcfc\" (UniqueName: \"kubernetes.io/projected/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-api-access-2mcfc\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.910939 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.910924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.911059 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.910949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19787062-9e22-44ed-b92f-46e8ddd9410b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.911248 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.911231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19787062-9e22-44ed-b92f-46e8ddd9410b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.911545 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.911525    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.911647 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.911581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.913362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.913343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19787062-9e22-44ed-b92f-46e8ddd9410b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:42.921297 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:42.921278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcfc\" (UniqueName: \"kubernetes.io/projected/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-api-access-2mcfc\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:43.072324 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:43.072292    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:43.208725 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:43.208698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"]
Jul 09 18:28:43.210720 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:28:43.210690    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19787062_9e22_44ed_b92f_46e8ddd9410b.slice/crio-170507f1751441705d06e5f9e3e4059c05474ccc054ec372ce1ba3927a377278 WatchSource:0}: Error finding container 170507f1751441705d06e5f9e3e4059c05474ccc054ec372ce1ba3927a377278: Status 404 returned error can't find the container with id 170507f1751441705d06e5f9e3e4059c05474ccc054ec372ce1ba3927a377278
Jul 09 18:28:44.090662 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:44.090611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerStarted","Data":"e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2"}
Jul 09 18:28:44.091129 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:44.090676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerStarted","Data":"170507f1751441705d06e5f9e3e4059c05474ccc054ec372ce1ba3927a377278"}
Jul 09 18:28:48.106850 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:48.106817    2562 generic.go:356] "Generic (PLEG): container finished" podID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerID="e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2" exitCode=0
Jul 09 18:28:48.107236 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:48.106865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerDied","Data":"e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2"}
Jul 09 18:28:49.113484 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:49.113446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerStarted","Data":"300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33"}
Jul 09 18:28:49.113857 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:49.113529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerStarted","Data":"61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf"}
Jul 09 18:28:49.113857 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:49.113750    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:49.143914 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:49.143849    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" podStartSLOduration=3.01243222 podStartE2EDuration="7.143828927s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.131396707s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:49.14337014 +0000 UTC m=+1515.114098038" watchObservedRunningTime="2026-07-09 18:28:49.143828927 +0000 UTC m=+1515.114556831"
Jul 09 18:28:50.117260 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:50.117226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:50.118683 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:50.118657    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 18:28:51.121206 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:51.121153    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 18:28:56.125593 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:56.125508    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:28:56.125985 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:28:56.125960    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 09 18:29:06.126749 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:06.126714    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:29:12.816134 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:12.816087    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"]
Jul 09 18:29:12.818528 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:12.816348    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container" containerID="cri-o://61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf" gracePeriod=30
Jul 09 18:29:12.818528 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:12.816427    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kube-rbac-proxy" containerID="cri-o://300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33" gracePeriod=30
Jul 09 18:29:13.204174 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.204132    2562 generic.go:356] "Generic (PLEG): container finished" podID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerID="300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33" exitCode=2
Jul 09 18:29:13.204347 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.204209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerDied","Data":"300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33"}
Jul 09 18:29:13.509159 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.509127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:13.659202 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.659182    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:29:13.745629 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.745541    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mcfc\" (UniqueName: \"kubernetes.io/projected/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-api-access-2mcfc\") pod \"19787062-9e22-44ed-b92f-46e8ddd9410b\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") "
Jul 09 18:29:13.745629 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.745576    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19787062-9e22-44ed-b92f-46e8ddd9410b-proxy-tls\") pod \"19787062-9e22-44ed-b92f-46e8ddd9410b\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") "
Jul 09 18:29:13.745629 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.745625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-cabundle-cert\") pod \"19787062-9e22-44ed-b92f-46e8ddd9410b\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") "
Jul 09 18:29:13.745912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.745644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19787062-9e22-44ed-b92f-46e8ddd9410b-kserve-provision-location\") pod \"19787062-9e22-44ed-b92f-46e8ddd9410b\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") "
Jul 09 18:29:13.745912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.745682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-rbac-proxy-sar-config\") pod \"19787062-9e22-44ed-b92f-46e8ddd9410b\" (UID: \"19787062-9e22-44ed-b92f-46e8ddd9410b\") "
Jul 09 18:29:13.746039 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.746013    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19787062-9e22-44ed-b92f-46e8ddd9410b" (UID: "19787062-9e22-44ed-b92f-46e8ddd9410b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:29:13.746136 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.746087    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19787062-9e22-44ed-b92f-46e8ddd9410b" (UID: "19787062-9e22-44ed-b92f-46e8ddd9410b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:29:13.746614 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.746586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19787062-9e22-44ed-b92f-46e8ddd9410b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19787062-9e22-44ed-b92f-46e8ddd9410b" (UID: "19787062-9e22-44ed-b92f-46e8ddd9410b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:29:13.747912 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.747888    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-api-access-2mcfc" (OuterVolumeSpecName: "kube-api-access-2mcfc") pod "19787062-9e22-44ed-b92f-46e8ddd9410b" (UID: "19787062-9e22-44ed-b92f-46e8ddd9410b"). InnerVolumeSpecName "kube-api-access-2mcfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:29:13.748004 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.747885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19787062-9e22-44ed-b92f-46e8ddd9410b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19787062-9e22-44ed-b92f-46e8ddd9410b" (UID: "19787062-9e22-44ed-b92f-46e8ddd9410b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:29:13.847010 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.846975    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-cabundle-cert\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:29:13.847010 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.847008    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19787062-9e22-44ed-b92f-46e8ddd9410b-kserve-provision-location\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:29:13.847010 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.847018    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:29:13.847461 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.847028    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mcfc\" (UniqueName: \"kubernetes.io/projected/19787062-9e22-44ed-b92f-46e8ddd9410b-kube-api-access-2mcfc\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:29:13.847461 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:13.847038    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19787062-9e22-44ed-b92f-46e8ddd9410b-proxy-tls\") on node \"ip-10-0-142-14.ec2.internal\" DevicePath \"\""
Jul 09 18:29:14.209759 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.209716    2562 generic.go:356] "Generic (PLEG): container finished" podID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerID="61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf" exitCode=0
Jul 09 18:29:14.209759 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.209757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerDied","Data":"61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf"}
Jul 09 18:29:14.209931 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.209777    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"
Jul 09 18:29:14.209931 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.209815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs" event={"ID":"19787062-9e22-44ed-b92f-46e8ddd9410b","Type":"ContainerDied","Data":"170507f1751441705d06e5f9e3e4059c05474ccc054ec372ce1ba3927a377278"}
Jul 09 18:29:14.209931 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.209846    2562 scope.go:117] "RemoveContainer" containerID="300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33"
Jul 09 18:29:14.219326 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.219123    2562 scope.go:117] "RemoveContainer" containerID="61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf"
Jul 09 18:29:14.226724 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.226707    2562 scope.go:117] "RemoveContainer" containerID="e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2"
Jul 09 18:29:14.233582 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.233556    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"]
Jul 09 18:29:14.233811 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.233799    2562 scope.go:117] "RemoveContainer" containerID="300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33"
Jul 09 18:29:14.234069 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:29:14.234046    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33\": container with ID starting with 300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33 not found: ID does not exist" containerID="300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33"
Jul 09 18:29:14.234165 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.234082    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33"} err="failed to get container status \"300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33\": rpc error: code = NotFound desc = could not find container \"300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33\": container with ID starting with 300e8b4256f9775f8581676d85ec469067c94b2104d90281c1cd4d1c0d794a33 not found: ID does not exist"
Jul 09 18:29:14.234165 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.234125    2562 scope.go:117] "RemoveContainer" containerID="61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf"
Jul 09 18:29:14.234369 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:29:14.234353    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf\": container with ID starting with 61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf not found: ID does not exist" containerID="61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf"
Jul 09 18:29:14.234405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.234375    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf"} err="failed to get container status \"61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf\": rpc error: code = NotFound desc = could not find container \"61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf\": container with ID starting with 61cd3366cb966afa5974cf07d81d5b12ec6071b15572f84adf3ba43a931581bf not found: ID does not exist"
Jul 09 18:29:14.234405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.234390    2562 scope.go:117] "RemoveContainer" containerID="e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2"
Jul 09 18:29:14.234595 ip-10-0-142-14 kubenswrapper[2562]: E0709 18:29:14.234577    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2\": container with ID starting with e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2 not found: ID does not exist" containerID="e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2"
Jul 09 18:29:14.234639 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.234603    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2"} err="failed to get container status \"e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2\": rpc error: code = NotFound desc = could not find container \"e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2\": container with ID starting with e27f0775c913b2419779d9dab4c6467a5903ecac669dcdabdc405d822d13ceb2 not found: ID does not exist"
Jul 09 18:29:14.237405 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.237386    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-gqpqs"]
Jul 09 18:29:14.562996 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:14.562910    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" path="/var/lib/kubelet/pods/19787062-9e22-44ed-b92f-46e8ddd9410b/volumes"
Jul 09 18:29:19.250042 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:19.250008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:19.347710 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:19.347677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:32.369007 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:32.368949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:47.739795 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.739764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:47.866336 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866309    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n4b7h/must-gather-gstws"]
Jul 09 18:29:47.866575 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866563    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kube-rbac-proxy"
Jul 09 18:29:47.866617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866577    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kube-rbac-proxy"
Jul 09 18:29:47.866617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866590    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="storage-initializer"
Jul 09 18:29:47.866617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866596    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="storage-initializer"
Jul 09 18:29:47.866617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866608    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container"
Jul 09 18:29:47.866617 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866613    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container"
Jul 09 18:29:47.866773 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866657    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kserve-container"
Jul 09 18:29:47.866773 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.866664    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19787062-9e22-44ed-b92f-46e8ddd9410b" containerName="kube-rbac-proxy"
Jul 09 18:29:47.870924 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.870906    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:47.873883 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.873868    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n4b7h\"/\"default-dockercfg-tj5tb\""
Jul 09 18:29:47.875153 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.875138    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4b7h\"/\"kube-root-ca.crt\""
Jul 09 18:29:47.875212 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.875165    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n4b7h\"/\"openshift-service-ca.crt\""
Jul 09 18:29:47.881474 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.881452    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4b7h/must-gather-gstws"]
Jul 09 18:29:47.884118 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.884083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tm2\" (UniqueName: \"kubernetes.io/projected/51626555-0c50-46ef-aa21-d53b6253e35d-kube-api-access-69tm2\") pod \"must-gather-gstws\" (UID: \"51626555-0c50-46ef-aa21-d53b6253e35d\") " pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:47.884256 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.884143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51626555-0c50-46ef-aa21-d53b6253e35d-must-gather-output\") pod \"must-gather-gstws\" (UID: \"51626555-0c50-46ef-aa21-d53b6253e35d\") " pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:47.885295 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.885270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:47.962040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.962017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:47.985227 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.985194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tm2\" (UniqueName: \"kubernetes.io/projected/51626555-0c50-46ef-aa21-d53b6253e35d-kube-api-access-69tm2\") pod \"must-gather-gstws\" (UID: \"51626555-0c50-46ef-aa21-d53b6253e35d\") " pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:47.985326 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.985258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51626555-0c50-46ef-aa21-d53b6253e35d-must-gather-output\") pod \"must-gather-gstws\" (UID: \"51626555-0c50-46ef-aa21-d53b6253e35d\") " pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:47.985557 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.985537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51626555-0c50-46ef-aa21-d53b6253e35d-must-gather-output\") pod \"must-gather-gstws\" (UID: \"51626555-0c50-46ef-aa21-d53b6253e35d\") " pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:47.996578 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:47.996531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tm2\" (UniqueName: \"kubernetes.io/projected/51626555-0c50-46ef-aa21-d53b6253e35d-kube-api-access-69tm2\") pod \"must-gather-gstws\" (UID: \"51626555-0c50-46ef-aa21-d53b6253e35d\") " pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:48.179398 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:48.179366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4b7h/must-gather-gstws"
Jul 09 18:29:48.305773 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:48.305744    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4b7h/must-gather-gstws"]
Jul 09 18:29:48.309883 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:29:48.309855    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51626555_0c50_46ef_aa21_d53b6253e35d.slice/crio-366b97ceb1b89bca1926bd5ed5543eb5920f7e1dc48bd4ce7a6de4ae6b5e9f2c WatchSource:0}: Error finding container 366b97ceb1b89bca1926bd5ed5543eb5920f7e1dc48bd4ce7a6de4ae6b5e9f2c: Status 404 returned error can't find the container with id 366b97ceb1b89bca1926bd5ed5543eb5920f7e1dc48bd4ce7a6de4ae6b5e9f2c
Jul 09 18:29:48.312028 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:48.312010    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:29:48.328669 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:48.328641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4b7h/must-gather-gstws" event={"ID":"51626555-0c50-46ef-aa21-d53b6253e35d","Type":"ContainerStarted","Data":"366b97ceb1b89bca1926bd5ed5543eb5920f7e1dc48bd4ce7a6de4ae6b5e9f2c"}
Jul 09 18:29:49.335527 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:49.335374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4b7h/must-gather-gstws" event={"ID":"51626555-0c50-46ef-aa21-d53b6253e35d","Type":"ContainerStarted","Data":"1c40fbb4b1490a0f5222a03c6ee800bd5a5fe940f3a9ceaacf0474a66c6bf9bb"}
Jul 09 18:29:49.335527 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:49.335432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4b7h/must-gather-gstws" event={"ID":"51626555-0c50-46ef-aa21-d53b6253e35d","Type":"ContainerStarted","Data":"2c1e4cabdc207eb6971eec38ad1ab0f8f8625d53bd6fab85942742408e965e34"}
Jul 09 18:29:49.357891 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:49.357276    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n4b7h/must-gather-gstws" podStartSLOduration=1.657958421 podStartE2EDuration="2.357256421s" totalImagesPullingTime="699.298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:49.356458522 +0000 UTC m=+1575.327186420" watchObservedRunningTime="2026-07-09 18:29:49.357256421 +0000 UTC m=+1575.327984319"
Jul 09 18:29:50.453393 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:50.453358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mb25c_1cb5fe9a-16d5-4bb5-a4cc-f62027c9b710/global-pull-secret-syncer/0.log"
Jul 09 18:29:50.626826 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:50.626779    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log"
Jul 09 18:29:50.778522 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:50.778427    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-14.ec2.internal_e946119a7cd2851de36f572276dc8964/haproxy/0.log"
Jul 09 18:29:53.702546 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:53.702516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:53.802668 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:53.802611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96g7t_1aab7f7d-2c9f-4812-b5b7-7b07d7151eb5/konnectivity-agent/0.log" limit=1024
Jul 09 18:29:55.357668 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:55.357605    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w54q8_445f8337-c953-4736-9a22-c89a9d895e61/node-exporter/0.log"
Jul 09 18:29:55.388478 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:55.388420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w54q8_445f8337-c953-4736-9a22-c89a9d895e61/kube-rbac-proxy/0.log"
Jul 09 18:29:55.422901 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:55.422873    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w54q8_445f8337-c953-4736-9a22-c89a9d895e61/init-textfile/0.log"
Jul 09 18:29:58.089311 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.089274    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"]
Jul 09 18:29:58.093084 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.093053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.105521 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.105496    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"]
Jul 09 18:29:58.167040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.166778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-sys\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.167040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.166824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-lib-modules\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.167040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.166853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-proc\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.167040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.166915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzbl\" (UniqueName: \"kubernetes.io/projected/406cc852-9803-47d6-bed0-4d6444e5fab0-kube-api-access-dmzbl\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.167040 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.166962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-podres\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267352 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzbl\" (UniqueName: \"kubernetes.io/projected/406cc852-9803-47d6-bed0-4d6444e5fab0-kube-api-access-dmzbl\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-podres\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267543 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-sys\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267657 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-podres\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267657 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-sys\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267764 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-lib-modules\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267852 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-lib-modules\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267852 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-proc\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.267946 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.267910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/406cc852-9803-47d6-bed0-4d6444e5fab0-proc\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.280798 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.280763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzbl\" (UniqueName: \"kubernetes.io/projected/406cc852-9803-47d6-bed0-4d6444e5fab0-kube-api-access-dmzbl\") pod \"perf-node-gather-daemonset-6ct2g\" (UID: \"406cc852-9803-47d6-bed0-4d6444e5fab0\") " pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.412362 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.412330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:58.574470 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:58.574444    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"]
Jul 09 18:29:58.576520 ip-10-0-142-14 kubenswrapper[2562]: W0709 18:29:58.576489    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod406cc852_9803_47d6_bed0_4d6444e5fab0.slice/crio-3d108c5f3db68c65788a52dc119bc05f7d68bb32ad599d1453cff687bb7ca9f8 WatchSource:0}: Error finding container 3d108c5f3db68c65788a52dc119bc05f7d68bb32ad599d1453cff687bb7ca9f8: Status 404 returned error can't find the container with id 3d108c5f3db68c65788a52dc119bc05f7d68bb32ad599d1453cff687bb7ca9f8
Jul 09 18:29:59.386697 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:59.386586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g" event={"ID":"406cc852-9803-47d6-bed0-4d6444e5fab0","Type":"ContainerStarted","Data":"2196de0a77cf87e377dfe705babe41cddbef1245d4b33af29e8b74cdd3365134"}
Jul 09 18:29:59.386697 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:59.386664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g" event={"ID":"406cc852-9803-47d6-bed0-4d6444e5fab0","Type":"ContainerStarted","Data":"3d108c5f3db68c65788a52dc119bc05f7d68bb32ad599d1453cff687bb7ca9f8"}
Jul 09 18:29:59.387461 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:59.387383    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:29:59.410775 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:29:59.410727    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g" podStartSLOduration=1.4107136599999999 podStartE2EDuration="1.41071366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:59.408975939 +0000 UTC m=+1585.379703840" watchObservedRunningTime="2026-07-09 18:29:59.41071366 +0000 UTC m=+1585.381441557"
Jul 09 18:30:00.143314 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:00.143287    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g8mbn_0fbee485-1e9f-4125-98e3-c4af2363faa5/dns/0.log"
Jul 09 18:30:00.189768 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:00.189742    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g8mbn_0fbee485-1e9f-4125-98e3-c4af2363faa5/kube-rbac-proxy/0.log"
Jul 09 18:30:00.268905 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:00.268881    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j45n7_11a66269-f70d-45a4-9bf4-0599903b8af8/dns-node-resolver/0.log"
Jul 09 18:30:00.888570 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:00.888544    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvsm2_13cb5c00-60d1-4a33-8302-cac1198f9c1f/node-ca/0.log"
Jul 09 18:30:02.259620 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:02.259591    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jfxn2_8a07e1f6-d288-4d69-bab9-05bd8087a18a/serve-healthcheck-canary/0.log"
Jul 09 18:30:02.912415 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:02.912387    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbfxf_c835b3ab-3208-4ee4-ae39-5ebc772e92e4/kube-rbac-proxy/0.log"
Jul 09 18:30:02.941986 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:02.941957    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbfxf_c835b3ab-3208-4ee4-ae39-5ebc772e92e4/exporter/0.log"
Jul 09 18:30:02.971714 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:02.971690    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbfxf_c835b3ab-3208-4ee4-ae39-5ebc772e92e4/extractor/0.log"
Jul 09 18:30:05.269291 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.269259    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-6h5j2_31c22e81-ed30-4bb7-999c-ba5b12eb950f/manager/0.log"
Jul 09 18:30:05.305831 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.305804    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gtczz_73420cbd-4486-4a2a-afaf-abe57464ebf5/s3-init/0.log"
Jul 09 18:30:05.331803 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.331762    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gtczz_73420cbd-4486-4a2a-afaf-abe57464ebf5/download-hf-model/0.log"
Jul 09 18:30:05.368890 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.368864    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rq9rh_97871dfd-e6ef-4bc0-8402-e21ad0c5feb5/s3-tls-init-custom/0.log"
Jul 09 18:30:05.400655 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.400626    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rq9rh_97871dfd-e6ef-4bc0-8402-e21ad0c5feb5/download-hf-model/0.log"
Jul 09 18:30:05.464825 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.464796    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-jp4ph_be0c1b71-7af6-498f-af34-1da8e40fecf7/s3-tls-init-serving/0.log"
Jul 09 18:30:05.515209 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.515181    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-jp4ph_be0c1b71-7af6-498f-af34-1da8e40fecf7/download-hf-model/0.log"
Jul 09 18:30:05.569523 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:05.569450    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-vpxrz_ff5af5b1-86a3-43c8-a2c5-c4e4a797ddce/seaweedfs/0.log"
Jul 09 18:30:06.407439 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:06.407403    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n4b7h/perf-node-gather-daemonset-6ct2g"
Jul 09 18:30:11.605331 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.605298    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:30:11.634758 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.634694    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/egress-router-binary-copy/0.log"
Jul 09 18:30:11.668587 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.668561    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/cni-plugins/0.log"
Jul 09 18:30:11.697616 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.697588    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/bond-cni-plugin/0.log"
Jul 09 18:30:11.731587 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.731555    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/routeoverride-cni/0.log"
Jul 09 18:30:11.763462 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.763429    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/whereabouts-cni-bincopy/0.log"
Jul 09 18:30:11.797106 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:11.797077    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xqwq_ec79ba9f-c9c8-4c31-911f-85e59598901f/whereabouts-cni/0.log"
Jul 09 18:30:12.412806 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:12.412782    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pmgzp_6abe54c6-a745-48bf-9e63-660221669af7/kube-multus/0.log"
Jul 09 18:30:12.497945 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:12.497921    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-846sw_80b70a46-af4d-46ce-9ef4-443bac043d9d/network-metrics-daemon/0.log"
Jul 09 18:30:12.526653 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:12.526622    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-846sw_80b70a46-af4d-46ce-9ef4-443bac043d9d/kube-rbac-proxy/0.log"
Jul 09 18:30:13.857070 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.857042    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-controller/0.log"
Jul 09 18:30:13.887916 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.887887    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/1.log"
Jul 09 18:30:13.887916 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.887908    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovn-acl-logging/0.log"
Jul 09 18:30:13.912194 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.912171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/kube-rbac-proxy-node/0.log"
Jul 09 18:30:13.940355 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.940332    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:30:13.966396 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.966370    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/northd/0.log"
Jul 09 18:30:13.993110 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:13.993066    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/nbdb/0.log"
Jul 09 18:30:14.019512 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:14.019488    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/sbdb/0.log"
Jul 09 18:30:14.069144 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:14.069046    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrngm_3502d811-6ad3-4ac5-a954-b170184e50e4/ovnkube-controller/0.log"
Jul 09 18:30:15.246378 ip-10-0-142-14 kubenswrapper[2562]: I0709 18:30:15.246353    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vbhk9_412937f2-07b0-4abf-b8e2-2206ec74df0c/network-check-target-container/0.log"